METRON-939 Elasticsearch ES5 with Xshield client support #619

wardbekker · 2017-06-19T10:04:28Z

Contributor Comments

This is a work-in-progress fix for Elasticsearch ES5 with Xshield client support. I'm opening a pull request to work through the remaining issues with the team. To be clear, the aim is discussion only for now, not for actually pulling in the code. @jasper-k verified the patch is working against a ES5.x Xpack enabled cluster.

Open issues in this pull request:

To have Storm and Elasticsearch 5.x client happily working together it's needed to have a Storm version that is compiled with log4j-core en log4j-api versions 2.8. This is already merged in the Storm sourcecode (see https://issues.apache.org/jira/browse/STORM-2326), however this fix is not shipping with HDP2.5.x. This fix is shipping withing the HDP 2.6 release.
While the metron-elasticsearch unit and integration tests are working, the ElasticsearchDataPrunerTest from metron-datamanagement are not. It can't seem to initialize ElasticSearch classes (e.g. java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.common.ParseField). It seems to be related with this issue: transport client fails in uber jar - no shortHash elastic/elasticsearch#21627 . I could use some help with resolving this.
The ES server used in the integration test is not using Xpack authentication. I could not find any public reference on creating an x-pack authentication enabled server via their SDK . Xpack authentication is confirmed to work for the Storm Writer bolt, but no integration test can be added until this issue is resolved.
I've updated the index template for yaf as used in the integration tests to be ES5.x compatible. Other index templates need the same treatment
ES & Kibana Mpacks are not updated.

cestella · 2017-06-19T14:03:34Z

...test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java

-                .put("index.number_of_shards", 1)
-                .put("node.mode", "network")
-                .put("index.number_of_replicas", 1);
+                .put("transport.type", "netty4");


Can we parameterize this?

sorry, nvm, I retract. This is in the integration test component.

cestella

Thanks for the contribution!

cestella · 2017-06-19T14:05:36Z

...test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java

+
+            byte[] indexTemplate = new byte[0];
+            try {
+                indexTemplate = Files.readAllBytes(Paths.get("/Users/wbekker/metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template"));


This path should be relative.

cestella · 2017-06-19T14:05:50Z

...test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java

+            try {
+                indexTemplate = Files.readAllBytes(Paths.get("/Users/wbekker/metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/yaf_index.template"));
+            } catch (IOException e) {
+                e.printStackTrace();


Probably want to throw back up a Runtime Exception

cestella · 2017-06-19T14:06:31Z

...test/java/org/apache/metron/elasticsearch/integration/components/ElasticSearchComponent.java

+        try {
+            node.close();
+        } catch (IOException e) {
+            e.printStackTrace();


We either want to throw up the exception or at least log it with a logger, rather than stdout.

wardbekker · 2017-06-19T17:43:58Z

Thanks for the review @cestella I've pushed a commit fixing the reported issues. (Again, not for merging)

wardbekker · 2017-06-27T13:59:43Z

hey @cestella, @simonellistonball, see updated contributor notes. It's not ready for a official pull request, but this gives a good idea on the impact on the code for a working ES5.x implementation.

JonZeolla · 2017-07-05T16:22:24Z

@wardbekker Can you please merge master and deconflict?

wardbekker · 2017-07-05T16:53:25Z

@JonZeolla done.

simonellistonball · 2017-07-27T12:21:50Z

Seems like this is failing on some un-related temporary test failures. Can we get Travis kicked, and see what's left to do on this?

… transport client

justinleet · 2017-09-19T12:54:03Z

As a note, this ticket is slightly impacted by the metaalerts backend ticket (#734). The alerts field in the various templates should be removed and the search queries for meta alerts updated according to https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields, in order to allow for searches against metaalerts without having to have an alert field in each template.

nickwallen · 2018-02-10T18:30:09Z

This functionality was completed in #840. As mentioned in #840 this inspired much of that work. Is there anything else needed from this PR? If not, can you close this PR @wardbekker ?

Thanks

wardbekker · 2018-02-10T19:55:55Z

agreed @nickwallen

wardbekker added 10 commits June 13, 2017 17:40

Upped the ES version and made it compile

d69d86c

fixed jackson dependency

ba80aab

Using netty4 transport type

d529af6

Changed transport client to Xpack transport client

e29514c

PreBuiltXPackTransportClient working

e8412e0

remove uncommented code

ea29f15

Green on all elasticsearch integration tests

bfade63

changed template to ES 5.0 schema

34b5b8e

Integration test cleanup

3c2c5e9

using the shaded jar in data-management project

21a1a71

cestella reviewed Jun 19, 2017

View reviewed changes

throwing up exceptions and removed hardcoded path

69460be

JonZeolla mentioned this pull request Jun 20, 2017

METRON-508 Expand Elasticsearch templates to support the standard bro logs #586

Closed

5 tasks

jknulst and others added 5 commits June 21, 2017 12:55

WardB path 619 + profile HDP2.5.3.0

51221d2

resolved merge

5062d63

elastic-shaded working

671b7bb

Working Elastic5.x connection

b89e420

Improved logging on ElasticsearchWriter

ec4e050

resolved merge with apache master

f95ce12

wardbekker added 5 commits August 4, 2017 10:01

Brought up-to-date with master. ElasticsearchDao now also using Xpack…

2c6e4fb

… transport client

Fixed compilation issues

fc8833b

Fixed equality compare for new ES version

4b5ca9b

Merge remote-tracking branch 'apache/master' into elasticsearch

c561abb

made master changes compatible with ES5

b831135

mmiklavc mentioned this pull request Nov 10, 2017

METRON-939: Upgrade ElasticSearch and Kibana #840

Closed

17 tasks

wardbekker closed this Feb 10, 2018

METRON-939 Elasticsearch ES5 with Xshield client support #619

METRON-939 Elasticsearch ES5 with Xshield client support #619

Uh oh!

Conversation

wardbekker commented Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor Comments

Uh oh!

cestella Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

cestella Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

cestella left a comment

Choose a reason for hiding this comment

Uh oh!

cestella Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

cestella Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

cestella Jun 19, 2017

Choose a reason for hiding this comment

Uh oh!

wardbekker commented Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wardbekker commented Jun 27, 2017

Uh oh!

JonZeolla commented Jul 5, 2017

Uh oh!

wardbekker commented Jul 5, 2017

Uh oh!

simonellistonball commented Jul 27, 2017

Uh oh!

justinleet commented Sep 19, 2017

Uh oh!

nickwallen commented Feb 10, 2018

Uh oh!

wardbekker commented Feb 10, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wardbekker commented Jun 19, 2017 •

edited

Loading

wardbekker commented Jun 19, 2017 •

edited

Loading