METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API

[mmiklavc]

Contributor Comments

https://issues.apache.org/jira/browse/METRON-1834

This task has been a long time coming after having completed the ES upgrade in https://issues.apache.org/jira/browse/METRON-939. Motivation for completing this now is that Elasticsearch will be deprecating use of the TransportClient in v 7.x. This PR migrates the Elasticsearch client from TransportClient to the newer Java REST API.

1. https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html
2. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-overview.html
3. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html

This builds off and finishes work started by @cestella here - https://github.com/cestella/incubator-metron/tree/es_rest_client. I condensed his branch into 1 flattened commit and built on top of it in order to provide attribution.

I have a number of tasks I'm still working through, but I wanted to get the review process started. I've minimally validated X-Pack auth and will have some follow-up for SSL. Test plans and a breakdown of the changes will be soon to follow. For starters, full dev should continue to work as normal and you should see data flowing into indexes for bro, snort, and yaf. There are some additional changes to how this client will be configured, which I'll be documenting shortly. The new client does not take a Map of settings any longer now that it is leverage Apache HTTP Async Client https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-low-usage-dependencies.html under the hood. This meant choosing a set of properties to expose and doing a translation to the builder pattern under the hood. Again, I'll have a write-up of this in the migration guide and update the README's accordingly.

NOTE: This checks off 2 items from this follow-on list #840 (comment)

1. Fix Log4j logging problem - classpath issues
2. Migrate to new ES REST client

Per discussion in the Metron Slack channel, I will be updating the Jira ticket with a series of tasks to be completed prior to acceptance, including performance regression testing compared with the old API.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

• Have you included steps to reproduce the behavior or problem that is being changed or addressed?

• Have you included steps or a guide to how the change may be verified and tested manually?

• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

  mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
  

• Have you written or updated unit tests and or integration tests to verify your changes?

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

  cd site-book
  mvn site
  

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

[mmiklavc]

A recent merge with #1218 has caused a number of integration test failures. Looking into it.

[mmiklavc]

Note, I force pushed this branch with revised history after the revert done by @nickwallen on #1218. This is effectively a merge with the latest master minus 1218 changes while we resolve the latent issues with ES and doc ids and guids.

[nickwallen]

Are there unit tests specifically for this class? I don't see any, but maybe I missed.

[mmiklavc]

Not unit tests, but the integration tests definitely cover this bit.

[mmiklavc]

I'm happy to clarify variable and method names here, but I want to be careful about overdoing it on the javadoc. It gets stale, is not testable, and is far more often overlooked than the code itself. I'm all for javadoc on public API's.

[cestella]

Just chiming in here in support of taking some care around javadoc. I tend to agree that it goes stale and it can become misleading. I think the metric (and this is just personal opinion here) should be that any public API should be well javadoc'd and otherwise documented (e.g. the REST API) and we should seriously consider javadoc'ing classes which are used broadly.

I tend to prefer a compromise whereby the class itself has documentation around the intent and use of the class in Javadoc and the methods are named well and broken up well. From there, I'm ok if there's less effort put around javadoc of individual methods.

Just my $0.02

[mmiklavc]

Thanks for the feedback/viewpoint @cestella . I think @nickwallen has some good points, in particular about why/when to choose one client or the other. I'm adding some of that detail now, and am linking aggressively to the ES docs so that we're not simply reproducing their documentation. In general, I do NOT think we should be writing javadoc like this - https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html. It is certainly detailed, but arguably pedantic for our purposes. Oh, and we should also be documenting Stellar functions - which is a feature we already support via the annotations you added some time ago.

[nickwallen]

I get your high-level take on DRY. Sure, makes sense. And I'm certainly not asking for anything close to the level of detail in the Arrays example.

I just so often find myself tracing through code for 15 minutes to find answers to questions that could have been answered with one, well-placed comment. Spent most of the day in that situation actually. That being said, whatever middle ground you can find will likely suffice.

[mmiklavc]

I've started addressing some of the questions you had @nickwallen. Not quite finished yet, but the rest should be soon to follow. I'd like to clean up the way we handle the ElasticsearchClient and its instantiation a bit.

[ottobackwards]

Just to chime in, understanding why something was done is important in maintaining the code down the line. @mmiklavc will not be the only one working on this code forever, and as such there should be some documentation somewhere for areas, no matter where they are, to explain important implementation decisions and concepts. The code is the ultimate source of truth, so I would prefer there. As someone who has to go spelunking through undocumented metron source, almost all of which I did not write or review, I can testify to this being helpful and in fact necessary if going forward we want to avoid regressions.

Side effects of methods are important too.
I think we should require javadoc and checkstyle for it, and would be doing so if we could get to a state where it would build with it on.

[mmiklavc]

I think we're all on the same page @ottobackwards, @nickwallen, and @cestella . As I mentioned early on, this effort was somewhat unique as it was built off of some POC work from @cestella from a couple months back, and I wanted to enhance that work (e.g. addressing the TODO's, bugs, and configuration issues mentioned) while refactoring/changing as little as possible. That's on me for not having gotten out in front of polishing a few of the common/reusable pieces a bit more, so your comments are well received. Updates soon to follow.

[mmiklavc]

@ottobackwards @nickwallen - just to shed some additional light on the choices made here, take a look at the DAO classes. The new ElasticsearchClient Metron client class lays some foundation towards wrapping the ES API. What this PR explicitly does not do is completely abstract away the ES dependencies. I believe that's a task that we should perform, but I do not believe it makes sense to couple it with the task of swapping out the client.

[mmiklavc]

Adapted from #840 (comment)

Test Script

Run full dev and verify you see data populating the alerts UI.

Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).

Preliminaries

Setup env vars. I like to do something like the following:

echo export METRON_HOST=node1 >> /root/.bashrc && \
echo export HDP_HOME=/usr/hdp/current >> /root/.bashrc && \
echo export KAFKA_HOME=/usr/hdp/current/kafka-broker >> /root/.bashrc && \
export SOLR_VERSION="6.6.2" && \
echo export SOLR_VERSION="$SOLR_VERSION" >> /root/.bashrc && \
echo export SOLR_HOME="/var/solr/solr-\${SOLR_VERSION}" >> /root/.bashrc && \
echo export ELASTIC_HOME="/usr/share/elasticsearch" >> /root/.bashrc && \
echo export KIBANA_HOME="/usr/share/kibana" >> /root/.bashrc && \
echo export ZOOKEEPER=\${METRON_HOST}:2181 >> /root/.bashrc && \
echo export BROKERLIST=\${METRON_HOST}:6667 >> /root/.bashrc && \
echo export STORM_UI=http://\${METRON_HOST}:8744 >> /root/.bashrc && \
echo export ELASTIC=http://\${METRON_HOST}:9200 >> /root/.bashrc && \
echo export ES_HOST=http://\${METRON_HOST}:9200 >> /root/.bashrc && \
echo export KIBANA=http://\${METRON_HOST}:5000 >> /root/.bashrc && \
export METRON_VERSION="0.6.1" && \
echo export METRON_VERSION="$METRON_VERSION" >> /root/.bashrc && \
echo export METRON_HOME="/usr/metron/\${METRON_VERSION}" >> /root/.bashrc && \
source /root/.bashrc 

Deploy the dummy parser

• Edit $METRON_HOME/config/zookeeper/parsers/dummy.json:

{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"dummy"
}

• Create the dummy kafka topic:
  /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic dummy --partitions 1 --replication-factor 1
• Persist config changes: $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z node1:2181
• Start via $METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s dummy

Send dummy data through

• Edit ~/msg.json with the following content:

{ "guid" : "guid0", "sensor.type" : "dummy", "timestamp" : 100 }

• Send msg.json through to kafka via cat ~/msg.json | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic dummy
• Validate data has been written to the index:

curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' 

Test Case: Update via patch

• Patch the message in ES and create a new field 'project' by executing
  the following:

curl -u user:password -X PATCH --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
  "guid" : "guid0",
    "sensorType" : "dummy",
    "patch" : [
    {
      "op": "add"
        , "path": "/project"
        , "value": "metron"
    }
  ]
}' 'http://node1:8082/api/v1/update/patch'

• Validate that the message has a field 'project':

curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
  "_source" : [ "project" ]
}
'

Test Case: Update via replace

• Replace the message in ES and create a couple of modifications:
  • new field new_field == "brand new"
  • modified timestamp == 7
    Execute the following:

curl -u user:password -X POST --header 'Content-Type: application/json' --header 'Accept: */*' -d '{
     "guid" : "guid0",
     "sensorType" : "dummy",
     "replacement" : {
       "source:type": "dummy",
       "guid" : "guid0",
       "new_field" : "brand new",
       "timestamp" : 7
      }
   }' 'http://node1:8082/api/v1/update/replace'

• Validate that the message has a field 'new_field':

curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
  "_source" : [ "new_field", "timestamp" ]
}
'

Meta Alerts Test

Set Up Base Data

We're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data

curl 'node1:9200/_cat/indices?v'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
green  open   snort_index_2017.09.06.14   1   0        130            0    180.9kb        180.9kb
green  open   bro_index_2017.09.06.14     1   0        160            0    564.3kb        564.3kb
green  open   .kibana                     1   0         52            0     71.2kb         71.2kb
green  open   metaalert_index             1   0          6            0     62.3kb         62.3kb

In this case, we care about snort_index_2017.09.06.14 and metaalert_index. To make our lives easier, we'll add a couple of stripped down messages to the our snort index (Make sure to sub in the correct index name):

curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H 'Content-Type: application/json' -d'
{
  "msg": "snort test alert",
  "ip_dst_port": "8080",
  "ethsrc": "0A:00:27:00:00:00",
  "protocol": "TCP",
  "source:type": "snort",
  "ip_dst_addr": "192.168.66.121",
  "ip_src_addr": "192.168.66.1",
  "threat:triage:rules:0:score": 10,
  "timestamp": 1504708744000,
  "threat:triage:rules:0:reason": null,
  "threat:triage:score": 10,
  "is_alert": "true",
  "ip_src_port": "50187",
  "guid": "snort_test_1"
}
'

curl -XPUT 'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_2?pretty' -H 'Content-Type: application/json' -d'
{
  "msg": "snort test alert 2",
  "ip_dst_port": "8080",
  "ethsrc": "0A:00:27:00:00:00",
  "protocol": "TCP",
  "source:type": "snort",
  "ip_dst_addr": "192.168.66.121",
  "ip_src_addr": "192.168.66.1",
  "threat:triage:rules:0:score": 10,
  "timestamp": 1504708744000,
  "threat:triage:rules:0:reason": null,
  "threat:triage:score": 10,
  "is_alert": "true",
  "ip_src_port": "50187",
  "guid": "snort_test_2"
}
'

Create a Meta Alert

At this point, we'll group these alerts together. In Ambari, go to Metron -> Quick Links -> Swagger UI and go to the "Meta Alert Controller".
Click on the create endpoint, and we'll want to send this request (but with the appropriate index subbed in):

{
  "groups": [
    "group_one",
    "group_two"
  ],
  "guidToIndices": {
    "snort_test_1":"snort_index_2017.09.06.14",
    "snort_test_2":"snort_index_2017.09.06.14"
  }
}

Validate that the metaalert was created and looks good:

curl 'node1:9200/m*/_search?pretty'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "metaalerts",
      "_type" : "metaalert_doc",
      "_id" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
      "_score" : 1.0,
      "_timestamp" : 1504725383439,
      "_source" : {
        "average" : 10.0,
        "min" : 10.0,
        "median" : 10.0,
        "alert" : [ {
          "msg" : "snort test alert",
          "threat:triage:rules:0:reason" : null,
          "ip_dst_port" : "8080",
          "ethsrc" : "0A:00:27:00:00:00",
          "threat:triage:score" : 10,
          "is_alert" : "true",
          "protocol" : "TCP",
          "source:type" : "snort",
          "ip_dst_addr" : "192.168.66.121",
          "ip_src_port" : "50187",
          "guid" : "snort_test_1",
          "ip_src_addr" : "192.168.66.1",
          "threat:triage:rules:0:score" : 10,
          "timestamp" : 1504708744000
        }, {
          "msg" : "snort test alert 2",
          "threat:triage:rules:0:reason" : null,
          "ip_dst_port" : "8080",
          "ethsrc" : "0A:00:27:00:00:00",
          "threat:triage:score" : 10,
          "is_alert" : "true",
          "protocol" : "TCP",
          "source:type" : "snort",
          "ip_dst_addr" : "192.168.66.121",
          "ip_src_port" : "50187",
          "guid" : "snort_test_2",
          "ip_src_addr" : "192.168.66.1",
          "threat:triage:rules:0:score" : 10,
          "timestamp" : 1504708744000
        } ],
        "max" : 10.0,
        "threat:triage:score" : 20.0,
        "count" : 2,
        "guid" : "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
        "groups" : [ "group_one", "group_two" ],
        "sum" : 20.0,
        "status" : "active"
      }
    } ]
  }
}

Note that both sub alerts are present, the various counts are filled in, a GUID has been given specifically to this meta alert, etc.

Searching

Searching from the REST API works mostly as expected. After the above data has been created, use the search endpoint to run this query:

{
  "from": 0,
  "size": 2,
  "indices": [
    "*"
  ],
  "query": "guid=snort_test_2",
  "sort": [
    {
      "field": "_timestamp",
      "sortOrder": "ASC"
    }
  ]
}

The result should look similar to the following, and have the messages we'd expect (one from snort and one from metaalerts):

{
  "total": 2,
  "results": [
    {
      "id": "snort_test_2",
      "source": {
        "msg": "snort test alert 2",
        "threat:triage:rules:0:reason": null,
        "ip_dst_port": "8080",
        "ethsrc": "0A:00:27:00:00:00",
        "threat:triage:score": 10,
        "is_alert": "true",
        "protocol": "TCP",
        "source:type": "snort",
        "ip_dst_addr": "192.168.66.121",
        "ip_src_port": "50187",
        "guid": "snort_test_2",
        "ip_src_addr": "192.168.66.1",
        "threat:triage:rules:0:score": 10,
        "timestamp": 1504708744000
      },
      "score": 1,
      "index": "snort_index_2017.09.06.14"
    },
    {
      "id": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
      "source": {
        "average": 10,
        "min": 10,
        "median": 10,
        "alert": [
          {
            "msg": "snort test alert",
            "threat:triage:rules:0:reason": null,
            "ip_dst_port": "8080",
            "ethsrc": "0A:00:27:00:00:00",
            "threat:triage:score": 10,
            "is_alert": "true",
            "protocol": "TCP",
            "source:type": "snort",
            "ip_dst_addr": "192.168.66.121",
            "ip_src_port": "50187",
            "guid": "snort_test_1",
            "ip_src_addr": "192.168.66.1",
            "threat:triage:rules:0:score": 10,
            "timestamp": 1504708744000
          },
          {
            "msg": "snort test alert 2",
            "threat:triage:rules:0:reason": null,
            "ip_dst_port": "8080",
            "ethsrc": "0A:00:27:00:00:00",
            "threat:triage:score": 10,
            "is_alert": "true",
            "protocol": "TCP",
            "source:type": "snort",
            "ip_dst_addr": "192.168.66.121",
            "ip_src_port": "50187",
            "guid": "snort_test_2",
            "ip_src_addr": "192.168.66.1",
            "threat:triage:rules:0:score": 10,
            "timestamp": 1504708744000
          }
        ],
        "max": 10,
        "threat:triage:score": 20,
        "count": 2,
        "guid": "d414a9f2-7a03-4e47-9108-16b6fc800dd3",
        "groups": [
          "group_one",
          "group_two"
        ],
        "sum": 20,
        "status": "active"
      },
      "score": 1,
      "index": "metaalerts"
    }
  ]
}

[mmiklavc]

@nickwallen I updated the ElasticsearchSearchIntegrationTest a bit. I'm now using the new client to setup the bro/snort templates, create the indices, and load the data.

[nickwallen]

@mmiklavc Keep in mind that I already have the open PR that refactors the data load in those tests to use the ElasticsearchUpdateDao #1247. I will work through the conflicts after this goes in.

Are you happy with this now? Is it done done?

[mmiklavc]

@mmiklavc Keep in mind that I already have the open PR that refactors the data load in those tests to use the ElasticsearchUpdateDao #1247. I will work through the conflicts after this goes in.

Are you happy with this now? Is it done done?

This is good to go now - I just finished testing SSL.

[mmiklavc]

Testing Notes For X-Pack AND SSL (I know, gettin fancy):

1. Install X-Pack

   This will also install the certgen tool which can be used for generating certificates for SSL.

   /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
   

2. Setup X-Pack User

   username = xpack_client_user
   password = changeme
   role = superuser

   sudo /usr/share/elasticsearch/bin/x-pack/users useradd xpack_client_user -p changeme -r superuser
   

3. Setup cert using the ES certgen tools

   Run the certgen tool. You can store the certs in /etc/elasticsearch/ssl/certs.zip when prompted. Use "node1" as the instance name in fulldev because you'll want it to match your host. You can leave the IP and DNS details blank.

   /usr/share/elasticsearch/bin/x-pack/certgen --pass
   

   Extract the certs

   cd /etc/elasticsearch/ssl
   unzip certs.zip
   # I flattened all ca/certs so it looks as follows
   ls -1 /etc/elasticsearch/ssl
   ca.crt
   ca.key
   esnode.crt
   esnode.key
   

4. Setup Elasticsearch to use the certs

   https://www.elastic.co/guide/en/x-pack/5.6/ssl-tls.html

   1. Modify /etc/elasticsearch/elasticsearch.yml

      xpack.ssl.key:                     /etc/elasticsearch/ssl/esnode.key
      xpack.ssl.certificate:             /etc/elasticsearch/ssl/esnode.crt
      xpack.ssl.certificate_authorities: [ "/etc/elasticsearch/ssl/ca.crt" ]
      xpack.security.transport.ssl.enabled: true
      xpack.security.http.ssl.enabled: true
      

5. Setup the client truststore

   1. Import the Certificate Authority (CA).

      • Specify an alias of your choosing. I chose "elasticCA".

      • You'll also be prompted for a password, which must be at least 6 characters. I used "apachemetron".

      • When prompted to "Trust this certificate?" type yes and hit enter.

        keytool -import -alias elasticCA -file /etc/elasticsearch/ssl/ca.crt -keystore clienttruststore.jks
        

   2. Import the data node certificate.

      • Specify an alias of your choosing. In fulldev use "node1" as it will need to match your hostname.

      • Enter the password that you set when creating the truststore.

        keytool -importcert -keystore clienttruststore.jks -alias node1 -file /etc/elasticsearch/ssl/esnode.crt
        

   3. Put the truststore in the storm user home dir for the purpose of our tests.

      mv clienttruststore.jks /home/storm/
      chown storm:hadoop /home/storm/clienttruststore.jks
      

6. Configure the Elasticsearch client in Metron

   1. Load the passwords in HDFS for x-pack and the truststore

      echo changeme > /tmp/xpack-password
      echo apachemetron > /tmp/truststore-password
      sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/
      sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/
      sudo -u hdfs hdfs dfs -put /tmp/truststore-password /apps/metron/elasticsearch/
      sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/*
      

   2. Modify the Metron global config with the SSL and X-Pack properties

      • Pull down the latest global config

        $METRON_HOME/bin/zk_load_configs.sh -m PULL -o $METRON_HOME/config/zookeeper -z $ZOOKEEPER -f
        
        

      • Update the configuration by adding the es.client.settings for xpack and SSL.

        "es.client.settings" : {
            "xpack.username" : "xpack_client_user",
            "xpack.password.file" : "/apps/metron/elasticsearch/xpack-password",
            "ssl.enabled" : true,
            "keystore.type" : "jks",
            "keystore.path" : "/home/storm/clienttruststore.jks",
            "keystore.password.file" : "/apps/metron/elasticsearch/truststore-password"
        }
        

      • Push the changes to Zookeeper

        $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z $ZOOKEEPER
        # Confirm your changes are there
        $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c GLOBAL
        

7. Restart Elasticsearch

   Note: The custom properties were never, and still are not exposed via the Elasticsearch MPack. Restarting via Ambari will overwrite any changes you make manually.

   service elasticsearch start

   • You should be able to connect to Elasticsearch via SSL in your browser: https://node1:9200/_cat/health?v.
   • The Head plugin should also allow you to connect by slightly changing your URL to use https for SSL: https://node1:9200/

8. Get rid of existing indexes so that we can easily tell that new data is flowing into Elasticsearch.

   Note: Use "node1" for the hostname. Otherwise, the cert will fail, triumphantly announcing curl: (51) SSL: certificate subject name 'node1' does not match target host name 'localhost'.

   curl --cacert /etc/elasticsearch/ssl/ca.crt --user xpack_client_user:changeme -XDELETE "https://node1:9200/squid*"
   curl --cacert /etc/elasticsearch/ssl/ca.crt --user xpack_client_user:changeme -XDELETE "https://node1:9200/bro*"
   curl --cacert /etc/elasticsearch/ssl/ca.crt --user xpack_client_user:changeme -XDELETE "https://node1:9200/snort*"
   curl --cacert /etc/elasticsearch/ssl/ca.crt --user xpack_client_user:changeme -XDELETE "https://node1:9200/yaf*"
   

9. Restart the indexing topology

   Ambari > Metron > Summary > Metron Indexing, choose restart

[nickwallen]

+1 Big effort and many thanks for this. I ran this up in Full Dev and it worked like a charm.

[mmiklavc]

Thanks @nickwallen. I pushed one more commit to make a note of this Jira in Upgrading.md. I verified it looks correct in the site-book and checked it off on the Jira tasks as well. I guess just a quick look (it's like 5 lines) when you get a moment would be great.

The last remaining bit before I merge this will be sharing some results around the performance/regression testing performed by @anandsubbu.

[nickwallen]

@mmiklavc +1 remains. Addition to upgrade doc looks good. A user may just have to translate from old properties to new.

[anandsubbu]

The last remaining bit before I merge this will be sharing some results around the performance/regression testing performed by @anandsubbu.

I completed a set of performance comparison tests between the REST and Transport clients. Here's the update.

Note on the Testing Done

• The performance regression comparison tests are run on a multi-node VM-based environment by:
  a) Running a set of tests using the ES REST Client
  b) Swap out the REST client with the Transport client, and re-run the same set of tests.

Observations

1. The indexing rate numbers with the REST client is similar to that of the Transport client.
2. I had noticed storm rebalance error messages for the kafkaSpout for both REST and Transport client runs, which required to adjust the values of request.timeout.ms and session.timeout.ms in the ES flux file under 'kafkaProps'. The errors were resolved after making these changes.
3. For the REST client, I observed listener timeout exceptions with the default settings. I had to increase max retry and socket timeout values in global config as follows, in order to resolve the exceptions.

  "es.client.settings" : {
    "max.retry.timeout.millis" : "60000",
    "socket.timeout.millis" : "60000"
  }

4. For both cases, the kafkaSpout complete latency was seen to be on the higher side (~30 secs)
5. Lastly, I noticed that there were a number of failed tuples seen in the kafkaSpout of the RA Indexing topology. I am continuing to dig more into this, but since this was seen for both the REST and Transport client tests, I am suspecting it could be an anomaly with the test environment, or a different issue common to both REST/Transport clients.

Based on the above, it is seen that there is no immediate regression in performance b/w the REST and Transport client.

+1, great work @mmiklavc

[mmiklavc]

I'm about to merge this and wanted to give some added feedback. @anandsubbu and I made some additional changes to the tuning options, specifically setting topology.ackers.executors equal to the number of inbound Kafka topic partitions, and we were able to see runs with no failures.

[nickwallen]

Ship it!

[anandsubbu]

specifically setting topology.ackers.executors equal to the number of inbound Kafka topic partitions

And the number of Workers too. i.e.
No. of Kafka partitions = No. of workers (ra_indexing_workers) = No. of Ackers (ra_indexing_acker_executors)

In my test, I set this to 10 and saw no failed tuples in the kafkaSpout.

My +1 holds. Thanks @mmiklavc !