METRON-529 Allow Configurable Periods for Profiler Client #334

nickwallen · 2016-11-01T15:00:05Z

METRON-529

By default, the Profiler creates Profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the client API is expecting 15 minute periods, it will not be able to read data generated by a Profiler that has been configured with a 1 hour period.

The period duration can be configured in the Profiler by altering the Profiler topology's static properties file. The Stellar Client API currently provides no means to configure the period duration and defaults also to 15 minutes. This means that the Stellar Client API can only read profiles with a period duration of 15 minutes.

This PR addresses this limitation so that the Profiler Client can be configured to read profile data no matter what configuration settings were used to generate that data.

Testing

I tested this change in the "Quick Dev" environment. I used the Profiler to create a Profile using a 1 minute period. I then confirmed that I was unable to read the data using the Profiler client with the default client configuration. I then changed the client configuration to a 1 minute duration. I was then able to read the profile data as expected.

Steps

Ensure the Snort sensor and the Snort parser are running. Stop all other topologies and sensors.
Edit /usr/metron/0.2.1BETA/config/profiler.properties and change the period duration to 1 minute. Also, point the Profiler at the enrichments topic to read data directly from the Snort Parser.

profiler.input.topic=enrichments
profiler.period.duration=1
profiler.period.duration.units=MINUTES

Follow the Profiler's Getting Started instructions to start generating profile data.
Configure the client to read data with a 1 minute period. To do so, edit /usr/metron/0.2.1BETA/config/zookeeper/global.properties.

{
  "es.clustername": "metron",
  "es.ip": "node1",
  "es.port": "9300",
  "es.date.format": "yyyy.MM.dd.HH",

  "profiler.client.period.duration": "1",
  "profiler.client.period.duration.units": "MINUTES"
}

Push the previous global configuration change to Zookeeper.

bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z node1:2181

Open up the Stellar shell and use the Profile client to read the data.

[root@node1 0.2.1BETA]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, profiler.client.hbase.table=profiler, profiler.client.column.family=P, profiler.client.hbase.table.provider=org.apache.metron.hbase.HTableProvider, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES, profiler.client.salt.divisor=1000}
[Stellar]>>> PROFILE_GET('test','192.168.138.158', 2, 'DAYS')
Functions loaded, you may refer to functions now...
[161.0]

nickwallen · 2016-11-01T15:58:36Z

Go Travis, Go!

mattf-apache · 2016-11-01T17:21:00Z

metron-analytics/metron-profiler-client/README.md

+By default, the Profiler creates Profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the client API is expecting 15 minute periods, it will not be able to read data generated by a Profiler that has been configured with a 1 hour period.

-By default, the Profiler creates Profiles with a period duration of 15 minutes.  This means that data is accumulated, summarized and flushed every 15 minutes.  The Client API must also have knowledge of this duration to correctly retrieve the profile data.  If the client API is expected 15 minute periods, it will not be able to read data generated by a Profiler that has been configured with a 1 hour period.
+The period duration can be configured in the Profiler by altering the Profiler topology's static properties file. The Stellar Client API can be configured by setting the following properties in Metron's global configuration.


"Profiler topology's static properties file": Suggest adding default file name and location here, for reference,
eg, "(default location: /usr/metron/(metron-version)/config/profiler.properties)"

Sure. Makes sense.

mattf-apache · 2016-11-01T17:58:42Z

metron-analytics/metron-profiler-client/README.md

+| profiler.client.column.family         | The name of the HBase column family used to store profile data.                                                                    | Optional | P        |
+| profiler.client.hbase.table.provider  | The name of the HBaseTableProvider implementation class.                                                                           | Optional |          |
+| profiler.client.period.duration       | The duration of each profile period.  This value should be defined along with `profiler.client.period.duration.units`.             | Optional | 15       |
+| profiler.client.period.duration.units | The units used to specify the profile period duration.  This value should be defined along with `profiler.client.period.duration`. | Optional | MINUTES  |


Profiler topology only specifies window duration in seconds. Since this has to be the same, suggest just using seconds, and not allowing units.

In actuality, the Profiler does match the same scheme as the client (using duration and units). The docs for the Profiler were not updated. I will fix that now. Good catch.

mattf-apache · 2016-11-01T18:12:09Z

metron-analytics/metron-profiler-client/README.md

+| Key                                   | Description                                                                                                                        | Required | Default  |
+| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------- | -------- |
+| profiler.client.hbase.table           | The name of the HBase table used to store profile data.                                                                            | Optional | profiler |
+| profiler.client.column.family         | The name of the HBase column family used to store profile data.                                                                    | Optional | P        |


The hbase column family is not documented as a config option in profiler.properties, https://github.com/apache/incubator-metron/tree/master/metron-analytics/metron-profiler#user-content-topology-configuration
Whether that's a docs bug or a profiler implementation limitation, would this be a good time for that fix or enhancement too?
Is a variable TableProvider also an issue for profiler?

I will update the Profiler docs. Thanks

mattf-apache · 2016-11-01T18:32:16Z

...tron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java

+    int saltDivisor = Integer.parseInt(configuredSaltDivisor);
+    LOG.debug("profiler client: {}={}", PROFILER_SALT_DIVISOR, saltDivisor);
+
+    return new SaltyRowKeyBuilder(saltDivisor, duration, units);


If GetProfile attempts to read a profile with the wrong window duration, will the user get a reasonable error message? I was unable to find this in GetProfile.java or GetProfileTest.java.

Unlike the profiler, it seems to me that there may be need for multiple simultaneous clients for different users, some of which may be consuming historical profile data that was taken with different window duration (and other settings) than is currently in use at the time the client is run. Therefore, I suggest that GetProfile should also take an optional override set of config arguments (any or all of profiler.client.* , possibly excepting table.provider) so that clients accessing different data won't stomp on each other.

Define "wrong window duration"? Can you give me an example of what you're thinking of?

Per (2) That's a good thought. I'm not ignoring it. I just need to noodle on that a bit. Hmm.

(1) "wrong" == GetProfile attempts to read a profile using a window duration different from what the profile was written with.
So if the profile was written using 1-hr intervals, and the client attempts to read it using 15-minute intervals, does the user get a reasonable error message?

Unfortunately, the user does not get a reasonable error message. It is a little bit of a sticky wicket. I am sure its doable, but I just haven't thought of a way that I really like.

A ProfilerClient uses a RowKeyBuilder along with the profile name, entity, period duration, and date range to construct a set of row keys that meet the user's query (see HBaseProfilerClient:100).

If there are no 'hits' for that set of row keys, then the user gets nothing back. I'd need some way to distinguish between "there is no data" versus "there is data, but it is written with a different period duration".

I think it'd be a good bit of work to catch all the corner cases. Although, I'm totally open to suggestions. Maybe there is an easy way that I am not thinking about.

I had thought about the profiler having that limitation, I'm glad you opened the jira, and I definitely agree that can be deferred. Regarding the client, though, I'm thinking about this scenario:
(a) There are one or two clients doing on-going routine analysis of up-to-date profiles, perhaps they are updaters for ML modules;
(b) Suddenly a security event requires analyzing some historical profile data from 2 months ago, that was done with different profiles -- different table, different period, something.
We shouldn't need to turn off (a) in order to do (b).

I won't insist on this, because certainly what you've already done is a great improvement. But I'd still like to suggest that a small addition to the mods you've made in GetProfile would give this flexibility. Do you mind if I just whomp it up? Then you can look at it, and if you still don't like it, I'll shut up :-)

I think what you are saying makes total sense. It is a good idea. We should have it. I'd prefer to do it as a follow-on JIRA though. Hey, maybe you could follow-on with your 'whomp'. Ha.

The one problem in doing it is due to the signature of PROFILE_GET. It would be ideal to add the period duration, etc as arguments towards the end of the function. That way the user can decide to pass in those additional args like duration or not. I'd hate to require the user to pass it in.

The problem is that the last argument is effectively a var arg of the group names. That makes it difficult to do what you are asking. How do you think we could work around that?

Right. Because of that we can't just generalize the PROFILE_GET argument list to include this. I'm thinking a PROFILE_GET_CONF that has a single string arg before "groups", consisting of a comma-separated list of key=value pairs, each overriding the global client parameter of the same name.

PROFILE_GET_CONF has the whole implementation, essentially same as current but with each of your new parameters possibly coming from the override string instead of from the global config. PROFILE_GET becomes a stub, invoking PROFILE_GET_CONF with an empty override string.

Okay, let's open a new jira for this too (separate from METRON-530, because they are completely separable and this is higher priority), and get on with this PR.

Good idea. I opened METRON-532. Feel free to comment on that issue, change the title, change the description, whatever you feel is appropriate. Hopefully, I've captured the core idea.

mattf-apache · 2016-11-01T18:37:03Z

...cs/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java

+    HTableInterface table = provider.getTable(config, "profiler");
+
+    long when = System.currentTimeMillis() - TimeUnit.DAYS.toMillis(2);
+    ProfileMeasurement measure = new ProfileMeasurement("profile1", "192.168.66.121", when, 15, TimeUnit.MINUTES);


Running tests with interval set to 15 minutes will make unit testing and integration testing unreasonably long. Please use small numbers of seconds as duration setting for all tests.
Or is time flow being mocked here, and I missed it?

This doesn't actually run as part of the tests and does not consume 15 minutes. So many times I've had to load up profile data for demo purposes. This main() function just provides a simple means to load up a bunch of profile data in a single shot.

Got it. The client tests run almost instantly. This writes something for the client to read WITHOUT having to wait for several sampling intervals. Thanks.

nickwallen · 2016-11-01T20:12:07Z

...tron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java

   * A global property that defines the name of the HBaseTableProvider implementation class.
   */
-  public static final String PROFILER_HBASE_TABLE_PROVIDER = "profiler.hbase.table.provider";
+  public static final String PROFILER_HBASE_TABLE_PROVIDER = "hbase.provider.impl";


I changed this property to hbase.provider.impl to match what is defined in other places like Enrichment.

This isn't an objection, just a question: I'm not yet real familiar with the structure of the configuration tree in ZK. Is this "hbase.provider.impl" parameter stored in a part of the tree where it clearly ONLY applies to profiler? Otherwise I would prefer to leave it with a "profiler." prefix.

It's not really something that you'd want to specify on a live cluster. It's only really useful when testing. It allows you to throw in a mocked HBase table so you don't have to spin up an HBase mini cluster. I questioned if I should even document it since a user will likely never use it.

For example, EnrichmentIntegrationTest.java:98 or ProfilerIntegrationTest.java:266. Open to suggestions

Got it. No problem.

mattf-apache · 2016-11-01T21:32:36Z

metron-analytics/metron-profiler/README.md

+| profiler.period.duration.units | The units used to specify the profile period duration.  This value should be defined along with `profiler.period.duration`. |
 | profiler.hbase.salt.divisor  |  A salt is prepended to the row key to help prevent hotspotting.  This constant is used to generate the salt.  Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster.  |
 | profiler.hbase.table | The name of the HBase table that profiles are written to.  |
+| profiler.hbase.column.family | The column family used to store profiles. |


mattf-apache · 2016-11-02T18:18:24Z

I'm now a strong +1 on this (by inspection), subject to the usual tests. Really important piece of work, kudos to Nick!

cestella · 2016-11-03T13:07:48Z

This looks good, @nickwallen. +1 binding from me by inspection.

METRON-529 Allow Configurable Periods for Profiler Client

a902c2a

nickwallen closed this Nov 1, 2016

nickwallen reopened this Nov 1, 2016

mattf-apache suggested changes Nov 1, 2016

View reviewed changes

METRON-529 Corrected documentation

2816481

nickwallen commented Nov 1, 2016

View reviewed changes

mattf-apache reviewed Nov 1, 2016

View reviewed changes

asfgit closed this in 05d29e7 Nov 3, 2016

nickwallen deleted the METRON-529 branch June 5, 2017 18:55

METRON-529 Allow Configurable Periods for Profiler Client #334

METRON-529 Allow Configurable Periods for Profiler Client #334

Uh oh!

Conversation

nickwallen commented Nov 1, 2016

Testing

Steps

Uh oh!

nickwallen commented Nov 1, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nickwallen Nov 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattf-apache Nov 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattf-apache commented Nov 2, 2016

Uh oh!

cestella commented Nov 3, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nickwallen Nov 1, 2016 •

edited

Loading

mattf-apache Nov 1, 2016 •

edited

Loading