METRON-1378: Create a summarizer #879

cestella · 2017-12-21T21:43:52Z

Contributor Comments

We have a nice and generalized infrastructure for loading data into HBase and interacting with it via flatfile_loader.sh and ENRICHMENT_GET(). It is also useful to summarize a set of data into a static data structure, store it on HDFS and interact with it via stellar. To this end, to complement the flatfile_loader.sh, we should have a flatfile_summarizer.sh that, using the same extractor config, will process a flat file and output a serialized object.

The usecase for this is as follows:
Let's say that I have a static list of domains in the second column of a CSV, domains.csv, and I want to generate a bloom filter with those domains in them sans TLD.

I should be able to create a file called bloom.ser with the serialized bloom filter given the extractor config:

{
  "config" : {
    "columns" : {
       "rank" : 0,
       "domain" : 1
    },
    "value_transform" : {
       "domain" : "DOMAIN_REMOVE_TLD(domain)"
    },
    "value_filter" : "LENGTH(domain) > 0",
    "state_init" : "BLOOM_INIT()",
    "state_update" : {
       "state" : "BLOOM_ADD(state, domain)"
                     },
    "state_merge" : "BLOOM_MERGE(states)",
    "separator" : ","
  },
  "extractor" : "CSV"
}

Note, the associated stellar function OBJECT_GET is available in #880.

Testing Plan

We should run the test plan for #445 to ensure no regressions since 80% of this PR is just refactoring existing abstractions to reuse.

Write out a String Locally

We are going to take the top 10k alexa domains (saved as part of #445 's test plan to ~/top-10k.csv)

Keep a running sample of 20 samples per thread
At the end, merge the samples and get a random domain from the merged samples
Write out the sample

Test

Create a file ~/extractor_sample.json with the following contents:

{
  "config" : {
    "columns" : {
       "rank" : 0,
       "domain" : 1
    },
    "value_transform" : {
       "domain" : "DOMAIN_REMOVE_TLD(domain)"
    },
    "value_filter" : "LENGTH(domain) > 0",
    "state_init" : "SAMPLE_INIT(20)",
    "state_update" : {
       "state" : "SAMPLE_ADD(state, domain)"
                     },
    "state_merge" : "GET_FIRST(SAMPLE_GET(SAMPLE_MERGE(states, SAMPLE_INIT(1))))",
    "separator" : ","
  },
  "extractor" : "CSV"
}

Summarize via $METRON_HOME//bin/flatfile_summarizer.sh -i ~/top-10k.csv -o ~/sample.ser -e ./extractor_sample.json -p 5 -b 128
Execute hexdump -C ./sample.ser and ensure that there is a string in there. It may end or start with some non-ascii bytes at the beginning and end.
e.g.

[root@node1 ~]# hexdump -C ./sample.ser
00000000  03 01 37 63 66 6d 6e e6                           |..7cfmn.|
00000008
[root@node1 ~]# cat top-10k.csv | grep 7cfmn
4696,7cfmnf.top

Typosquatting Use-case Testing

You can also follow the testing plan for #882 as this code is merged into that PR and it shows how this feature can be used in a real use-case.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

Have you included steps to reproduce the behavior or problem that is being changed or addressed?
Have you included steps or a guide to how the change may be verified and tested manually?
Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh 
```
Have you written or updated unit tests and or integration tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
```
cd site-book
mvn site
```

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

nickwallen · 2017-12-24T15:55:44Z

IMHO, this seems really complex. That JSON configuration looks really scary to me and sadly I don't think I could bring myself to use this.

Fundamentally the need you are addressing is that you want to manipulate a text file and leverage Stellar as part of that pipeline, right?

In a perfect world, I'd be able to use Unix-y tools like Grep, Sed, Awk, etc and introduce Stellar expressions as part of the pipeline. I know it is already possible to some degree with the REPL.

I am wondering if with a little brainstorming we might be able to arrive at a simpler solution that allows us to leverage tried-and-true UNIX pipelines so that people can use the Unix-y tools they are familiar with. I feel we'll get better user traction with a simpler, more familiar approach.

ottobackwards · 2017-12-24T21:04:26Z

I think the need is to 'pre-create' complex objects, and re-use them across multiple stellar rule executions, with the bloom-filter being the example.

Is that close?

cestella · 2017-12-25T00:05:10Z

@nickwallen I definitely hear you, the JSON configs are more complex than I'd like. I'd like another more composable solution available using stellar functions available in the REPL.

The intent of this PR, however, was a minimal extension of our existing extractor config to enable this use-case. This PR isn't making a complex bit of JSON per se, but rather it's reusing our existing complex bit of JSON to enable the use-case. I would love to create a discuss thread to see about a more REPL-friendly approach to this, but I think that belongs as a follow-on. Thoughts?

Edit: I wanted to make sure that it was clear that this is just a reuse and very slight extension of the extractor config documented here

cestella · 2017-12-25T00:06:33Z

@ottobackwards Yes, that's spot on. It's to enable creation of summarization objects in a method similar to (and reusing the configs and infrastructure of) the flat file loader. The idea is that this is just one way to create these objects, we might (likely will) have more ways to create them in the future.

cestella · 2017-12-25T00:24:12Z

Also, a wizard-like UI could simplify this dramatically. That was one of the thoughts around extending and reusing the existing infrastructure in the first pass of this rather than creating a new way of iterating over flat files. I expect some users will want to use the UI and be frightened of composing stellar functions in the REPL.

cestella · 2017-12-25T00:53:27Z

After more consideration and more egg nog, I decided that I'd create a DISCUSS thread about this entire use-case. We can move the discussion there.

EDIT: I also, to make sure we had a full-throated discussion, included what I'd like to see as step 2, which I think is closer to what you were proposing @nickwallen. Interested to hear your thoughts there.

cestella · 2018-01-05T20:04:31Z

So, the discuss thread has been going for some time now and the discussion is mostly around forward-thinking extensions to this. Are we at the point to agree that this is a viable first step and this can be reviewed?

justinleet

Thanks for the contribution here, this is definitely good stuff. I took a first pass through the code and left a couple minor comments on the assumption that the dev thread has pretty much died down.

I still need to spin it up, and look forward to testing it out.

justinleet · 2018-01-08T15:49:48Z

...c/main/java/org/apache/metron/dataloads/nonbulk/flatfile/importer/AbstractLocalImporter.java

+    return LocationStrategy.getLocation(input, fs);
+  }
+
+  public void extractLineByLine( List<String> inputs


It's not something I'd want any work done on here, but have we put any thought into having this be a generic delimiter for lines, rather than just lines? It's probably just a light refactoring, but it's always really annoying to me when I need a non-newline delimiter and then end up not being able to trivially specify it. It's uncommon, but it does seem to pop up now and again.

justinleet · 2018-01-08T15:56:16Z

metron-platform/metron-data-management/README.md

+| -m         | --import_mode       | No           | The Import mode to use: LOCAL, MR.  Default: LOCAL                                                                                                                                  |
+| -om        | --output_mode       | No           | The Output mode to use: LOCAL, HDFS.  Default: LOCAL                                                                                                                                  |
+| -i         | --input             | Yes          | The input data location on local disk.  If this is a file, then that file will be loaded.  If this is a directory, then the files will be loaded recursively under that directory.  |
+| -i         | --output            | Yes          | The output data location.    |


-o here, not -i

justinleet · 2018-01-08T17:42:14Z

...ata-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/writer/Writer.java

+import java.util.Optional;
+
+public interface Writer {
+  void validate(Optional<String> output, Configuration hadoopConfig);


Why doesn't this either return something or throw a checked exception? Right now the validate method is just something that you call and it doesn't do anything except throw runtime exception in the impls, rather than giving the caller an explicit ability to do anything about the validation failure.

good catch.

justinleet · 2018-01-08T22:03:02Z

I spun this up in the context of the combined PR, and everything worked as advertised, barring the UI because of ES5 issues. I was able to validate that data flowed through as expected by querying against the indices directly.

I'm good with this being a viable first step, and am +1.

mmiklavc · 2018-01-08T23:24:57Z

@justinleet

barring the UI because of ES5 issues

What sort of issues?

justinleet · 2018-01-08T23:44:25Z

@mmiklavc Check out #882 (comment). Looks like the squid mapping @cestella uses doesn't line up (which isn't terribly surprising because it was ES2 until a couple hours ago).

cestella · 2018-01-09T19:53:29Z

Just following up, I have migrated the mapping of existing data to a template in the instructions and the type mismatch for ip_dst_addr is no longer an issue. The PR as it currently stands has the correct instructions and is ready for review to continue.

justinleet · 2018-01-10T16:04:33Z

I'm still +1 on this, thanks again.

cestella · 2018-01-12T16:21:10Z

Ok, @justinleet has given a +1, do we have any existing reservations after the discussion thread and the review here on this work? If not, then I'm going to commit on Monday.

ottobackwards · 2018-01-12T16:30:54Z

I'm not getting in front of the train on this.
I am neutral on this other than my comments to help the conversation and attempt to focus things.

I don't know how to enter "don't mind me, pretend i'm not here".

cestella · 2018-01-12T16:32:56Z

haha @ottobackwards neutral would be a +0, which is fine. Thanks for your constructive comments on the discuss thread and here. As always, they're much appreciated. :)

ottobackwards · 2018-01-12T16:41:44Z

+0

alinazemian · 2018-01-15T03:26:58Z

@cestella Is there any document or description regarding this feature? How would the performance be comparable with normal HBase enrichment?

nickwallen · 2018-01-16T15:23:37Z

+0 I'm sure what's here is solid, but I have not reviewed it myself. I just want to clear the way for this to get merged.

I don't necessarily like the usability of this approach, but I think it is a good first step. And merging this doesn't preclude providing alternative approaches or enhancing this approach for better usability later.

I do love the use case that was added that drove the need for this functionality. So good stuff @cestella!

cestella added 5 commits December 21, 2017 10:17

flatfile summarizer initial commit.

9c492c4

typo

1568114

Updating tests and docs.

935d4d2

more docs.

afe91c3

Renamed test.

d955e26

cestella mentioned this pull request Dec 21, 2017

METRON-1379: Add an OBJECT_GET stellar function #880

Closed

10 tasks

cestella added 3 commits December 22, 2017 07:23

Adding a ToString writer.

ac3c612

Renamed to console writer

34cdb55

newline issue.

b3e4408

cestella mentioned this pull request Dec 22, 2017

METRON-1380: Create a typosquatting use-case (commit after METRON-1379, METRON-1377, METRON-1378) #882

Closed

10 tasks

cestella added 2 commits December 23, 2017 10:32

Allowing empty outputs

767e497

Missed a compilation error.

b4e40a4

Merge branch 'master' into flatfile_object_gen

3ed0568

cestella closed this Jan 8, 2018

cestella reopened this Jan 8, 2018

justinleet reviewed Jan 8, 2018

View reviewed changes

cestella added 2 commits January 8, 2018 14:54

Reacting to Justin's comments.

fe8be62

Merge branch 'master' into flatfile_object_gen

6a6968e

asfgit closed this in 5d3e73a Jan 16, 2018

iraghumitra pushed a commit to iraghumitra/incubator-metron that referenced this pull request Feb 17, 2018

METRON-1378: Create a summarizer closes apache#879

b25de9c

METRON-1378: Create a summarizer #879

METRON-1378: Create a summarizer #879

Uh oh!

Conversation

cestella commented Dec 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor Comments

Testing Plan

Write out a String Locally

Test

Typosquatting Use-case Testing

Pull Request Checklist

For all changes:

For code changes:

For documentation related changes:

Note:

Uh oh!

nickwallen commented Dec 24, 2017

Uh oh!

ottobackwards commented Dec 24, 2017

Uh oh!

cestella commented Dec 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cestella commented Dec 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cestella commented Dec 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cestella commented Dec 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cestella commented Jan 5, 2018

Uh oh!

justinleet left a comment

Choose a reason for hiding this comment

Uh oh!

justinleet Jan 8, 2018

Choose a reason for hiding this comment

Uh oh!

justinleet Jan 8, 2018

Choose a reason for hiding this comment

Uh oh!

cestella Jan 8, 2018

Choose a reason for hiding this comment

Uh oh!

justinleet Jan 8, 2018

Choose a reason for hiding this comment

Uh oh!

cestella Jan 8, 2018

Choose a reason for hiding this comment

Uh oh!

justinleet commented Jan 8, 2018

Uh oh!

mmiklavc commented Jan 8, 2018

Uh oh!

justinleet commented Jan 8, 2018

Uh oh!

cestella commented Jan 9, 2018

Uh oh!

justinleet commented Jan 10, 2018

Uh oh!

cestella commented Jan 12, 2018

Uh oh!

ottobackwards commented Jan 12, 2018

Uh oh!

cestella commented Jan 12, 2018

Uh oh!

ottobackwards commented Jan 12, 2018

Uh oh!

alinazemian commented Jan 15, 2018

Uh oh!

nickwallen commented Jan 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

cestella commented Dec 21, 2017 •

edited

Loading

cestella commented Dec 25, 2017 •

edited

Loading

cestella commented Dec 25, 2017 •

edited

Loading

cestella commented Dec 25, 2017 •

edited

Loading

cestella commented Dec 25, 2017 •

edited

Loading