METRON-1878: Add Metron as a Knox service

[merrimanr]

Contributor Comments

This PR adds REST and the Alerts UI as services in Knox.

Changed Included

• Fixed minor path issues in Alerts UI code. Files include:
  • app.component.html - path should be relative
  • authentication.service.ts - logout should match other endpoints
  • alert-search.directive.ts - path should be relative
  • index.html - a relative base href means Knox does not have to rewrite links
• Fixed minor path issues in Management UI related code. Files include:
  • _fonts.scss - path should be relative
  • authentication.service.ts - logout should match other endpoints
  • ace-editor.component.ts - path should be relative
  • index.html - a relative base href means Knox does not have to rewrite links
• Fixed minor bug in Management UI that was sending the wrong headers when getting the current user (navbar.component.ts)
• Added AppConfigService to the Management UI (very similar to the work done in METRON-1875: Expose configurable global settings in the Alerts UI #1266):
  • Added app-config.json file and AppConfigService.ts service
  • Added code to initialize the app config in app.module.ts
  • Adjusted existing services and unit tests for this new pattern
• Created Metron Knox topologies and service definition files for REST and Alerts UI. The service definition files are relatively simple and should look similar to other services.
• Created a script to deploy Metron files to Knox
• Added a Spring "knox" profile that configures Swagger with the right endpoint paths and enables the Knox SSO ServletFilter.
• Added service definition files to rpm spec
• Added "apiRoot" and "loginPath" properties to the Alerts UI to format REST request urls and redirect to the right login page after logout.
• Added documentation and unit tests

Testing

This feature can be tested in full dev with some manual configuration. After spinning up full dev:

1. Add Knox as a Service in Ambari (Admin > Stacks and Versions > Add Service next to Knox)
2. Follow the instructions here to enable the Knox LDAP in full dev. You should be able to authentication with REST using guest/guest-password after this is done.
3. Install the Metron topology and service definition files by running the script at /usr/metron/0.7.0/bin/install_metron_knox.sh. This script copies the metron.xml and metronsso.xml topology files to /usr/hdp/current/knox-server/conf/topologies and the service definition files to /usr/hdp/current/knox-server/data/services/metron-rest/0.6.1 and /usr/hdp/current/knox-server/data/services/metron-alerts/0.7.0
4. Update the REST API path for the Alerts UI. Currently expressjs proxies REST requests from the Alerts UI but this will now be done by Knox instead. The apiRoot setting in app-config.json is used to construct the url for REST requests. Change this setting in /usr/metron/0.7.0/web/alerts-ui/assets/app-config.json from:

{
  "apiRoot": "/api/v1"
}

to:

{
  "apiRoot": "/gateway/metron/metron-rest/api/v1"
}

5. Update the login path for the Alerts UI. The Alerts UI normally has a dedicated login page but this is replaced with a Knox login page when Knox is enabled. Knox will intercept the request, redirect to the login page, then redirect back to the original requested url. Set this property to the home page of the Alerts UI. Change the loginPath setting in /usr/metron/0.7.0/web/alerts-ui/assets/app-config.json from:

{
  "loginPath": "/login"
}

to:

{
  "loginPath": "/gateway/metron/metron-alerts/"
}

Any changes to app-config.json require a page refresh.

6. Make the same changes for the Management UI. Change this setting in /usr/metron/0.7.0/web/alerts-ui/assets/app-config.json from:

{
  "apiRoot": "/api/v1",
  "loginPath": "/login"
}

to:

{
  "apiRoot": "/gateway/metron/metron-rest/api/v1",
  "loginPath": "/gateway/metron/metron-management/sensors"
}

7. Add the knox Spring active profile in Ambari. Navigate to Ambari > Metron > Configs > REST.
   Change the Active Spring profiles setting from dev to dev,knox. This sets the Swagger context path to Knox and adds the Knox SSO servlet filter.
8. Update the Spring options in Ambari. Get the Knox public key with the following command on full dev:

openssl s_client -connect node1:8443 < /dev/null | openssl x509 | grep -v 'CERTIFICATE' | paste -sd "" -

Copy the value generated from that command. Navigate to Ambari > Metron > Configs > REST and change the Metron Spring options to:

--knox.root=/gateway/metron/metron-rest --knox.sso.pubkey=<key generated from openssl command>

The knox.root setting sets the Knox root path. The Swagger interface is served by the REST application and provides links for the various endpoints. Swagger must be configured with the new Knox root path for REST. The knox.sso.pubkey setting sets the Knox public key that will be used to verify Knox SSO tokens.

9. Restart REST with Ambari and you should now be able to access both Swagger and the Alerts UI through Knox with SSO enabled. The REST and Alerts UI urls are now:

• Swagger - https://node1:8443/gateway/metron/metron-rest/swagger-ui.html
• Alerts UI - https://node1:8443/gateway/metron/metron-alerts/alerts-list
• Management UI - https://node1:8443/gateway/metron/metron-management/sensors

Everything should function normally including login, logout and session timeout. To test the SSO part:

1. Request the global config with this url in a browser:

https://node1:8443/gateway/metron/metron-rest/api/v1/global/config

You should be redirected to the Knox SSO url with the originalUrl param properly set:

https://node1:8443/gateway/knoxsso/knoxauth/login.html?originalUrl=https://node1:8443/gateway/metron/metron-rest/api/v1/global/config

2. After authenticating with LDAP credentials, you should be successfully redirected and see the global config in the browser.

3. To verify roles are working correctly, authenticate as the admin user with admin/admin-password. The https://node1:8443/gateway/metron/metron-rest/api/v1/user/whoami/roles endpoint should return ROLE_USER and ROLE_ADMIN just as it currently does.

4. Navigate to the Alerts UI:

https://node1:8443/gateway/metron/metron-alerts/alerts-list

You should go directly to the Alerts UI without being prompted for credentials.

5. Click the logout link in the Alerts UI and verify you are redirected to the Knox login page.

6. Login back into the Alerts UI and wait. The session should timeout and you should be redirected back to the login page. The session timeout can be changed in metronsso.xml with the `` property. To change from the default of 30 seconds to 5 minutes, change the property from this:

<param>
      <name>knoxsso.token.ttl</name>
      <value>30000</value>
    </param>

to this:

<param>
      <name>knoxsso.token.ttl</name>
      <value>300000</value>
    </param>

7. Perform steps 4-6 for the Management UI.

Outstanding Items

Installing Metron in Knox

Installing a service in Knox involves 2 steps:

1. Deploy the service definition files mentioned earlier by copying them to Knox directories on the machine where Knox is installed
2. Either update an existing Knox topology file or create a new topology file. This file configures a collection of services. In our case this includes authentication and urls to the services exposed.

What is the best way to install these files in Knox? The approach in this PR only works if Metron is installed on the same machine as the Knox gateway.

I also opted to create a dedicated Metron topology file. I think this will give us more control and allow us to expose properties in a more user-friendly way. Knox has a default topology that you can configure in Ambari but it's exposed as is: a big xml file. Does anyone think we should use the default topology file instead?

Adding Knox to the stack

This process requires that we add Knox separately through Ambari. Do we want to make Knox a dependency for Metron similar to Kafka, Storm, etc?

Does it make sense to require that Knox and Metron are colocated? Doing this simplifies installation and configuration and I would vote we make this a requirement unless there is a good reason not to. Does anyone think there are cases where users would need to install them on different machines? If we do that, how do we install the service definition files?

Ambari automation

Currently setting up Metron with Knox is a manual process. I believe all of this can easily be automated with Ambari. How far do we want to go in this initial pass? Do we want to just document it for now? If we did want to automate this setup with Ambari, we would need to:

• Put the app-config.json under Ambari control and expose the apiRoot setting
• Add the knox.root setting to the REST config file (rest_application.yml) and expose it through an Ambari setting
• Add an Ambari setting similar to the LDAP enabled setting that automatically adds the knox spring profile to the list of active profiles
• Possibly put the metron.xml topology file under Ambari control
• Add a call to install_metron_knox.sh in the appropriate Metron Mpack script

Quick Links

At this point I do not know how to make Ambari quick links dynamic. You can bind the port to a property but I don't know if you can do that for the scheme, host, and path. This means that after you follow these instructions for setting up Metron with Knox, the quick links will still point to the urls without Knox. Does anyone know how to change quick links to point to Knox automatically? There has been discussion of enabling Knox by default so in that case we can just set the quick links to the Knox urls.

Hopefully a working example will provide more context as we plan out this feature.

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

• Have you included steps to reproduce the behavior or problem that is being changed or addressed?

• Have you included steps or a guide to how the change may be verified and tested manually?

• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

  mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
  

• Have you written or updated unit tests and or integration tests to verify your changes?

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

  cd site-book
  mvn site
  

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

[justinleet]

Re: quick links; you might be out of luck, based on https://issues.apache.org/jira/browse/AMBARI-21325. There might be ways around it (or require manual updating), but beyond that I don't know.

[justinleet]

Thoughts on the outstanding items

Installing Metron in Knox

• re: "The approach in this PR only works if Metron is installed on the same machine as the Knox gateway." I know of Hadoop installs using multiple Knox instances with a load balancer in front of them. I would expect this to be a reasonable expectation for people with sufficiently large Metron installs (or many tenants or whatever). I expect that to influence the decision here.
• re: topology file. Can you give a bit more detail on the pros/cons? Is there any standard that other implementors are using or recommendations in the docs?

Adding Knox to the stack

• As noted above, multiple Knox instances with load balancer is a potential configuration users will want. In that case, colocation is not a given.
• I don't think Knox as a required service is a given. I wish Ambari let you have optional services and gate things more directly. We might be able to mimic that functionality by hiding the relevant configs unless Knox is enabled. It might still require the user to hit a toggle, but it might be possible to check against a Knox service config directly for that. I've never tried it (and it might be nice to do in general for some of our other configs, e.g. Solr and ES).

Ambari Automation

• I'm inclined to consider Ambari necessary for effort as a whole to be really done. I'd prefer it to be done now just to have it done and coherent, but I'm fine with it being a follow-on PR (given the usual assumptions that everything continues to work as expected and is documented, otherwise I'd be inclined to use feature branch). It means docs are throwaway once that goes in, but I'm personally fine with it being manually documented then automated later.

Quick Links

• As mentioned above, I think we're out of luck here.

[merrimanr]

Thanks for the feedback @justinleet.

• Colocating: You bring up a good point about multiple Knox instances. The primary challenge here is that enabling Metron on Knox requires copying files to local directories on each Knox Gateway. If Knox exposed this through a REST API or could read files from HDFS this would not be an issue but I'm not aware that either of those options are supported. This is not an issue with other services (HDFS, HBase, etc) because those services have been added to the Knox project and are installed along with Knox. We could go that route but there are some major drawbacks (see the "Knox SSO feature branch review and features" discussion thread). It's not clear to me how we solve this in an automated way. Is this (copying Metron files to Knox Gateway hosts) possible with Ambari?

• Custom topology vs Knox default: There are several advantages to maintaining our own topology file:

  • We have complete control over how we expose settings in Ambari. The Knox default topology is exposed in Ambari as raw xml.
  • We have control over the url value where the Knox default url is /gateway/default/<service name>. There may be a way to change this but I don't know how.
  • The topology file will be smaller and easier to understand because it only contains Metron services.

  The main disadvantage I see is related to the issue described above: we need a way to copy this
  file to Knox Gateway hosts and update it.

• Ambari automation: I do believe everything still works as before in this PR. I think some decisions we make here will influence that work so a follow on seems appropriate to me.

• Quick Links: I agree, looks like we're out of luck here. I think we'll either need to make Knox the default and make the Metron quick links match or live with quick links being updated manually.

[justinleet]

I'm not sure if we can have Ambari do that in a general way. Right now we do have the Metron client get distributed throughout for the purpose of setting up Storm security. We could also do the same thing for whatever we need to distribute for Knox. It means a Metron Client needs to coexist with any Knox servers, but that seems like a reasonable restriction (even if I'd rather not have to deal with it). I think that would also address the custom topology question.

If we're okay with putting more load on the Metron Client, I'd rather use the custom topology and just do it that way.

For quicklinks, I'm okay with having the manual update process documented as part of the Knox setup, as long as it's noted its not a Metron issue, it's an Ambari/Knox interaction issue (and maybe just link to the Jira)

[mmiklavc]

@merrimanr Where does this leave the management UI if I enable Knox?

[mmiklavc]

I will be looking into this further, but I see a lot of the following in the management UI logs. This repeated for more than 80k lines in a matter of minutes:

18/12/06 20:13:23 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:23 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb0099 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:23 ERROR controller.RestExceptionHandler: Encountered error: Unable to get column metadata
org.apache.metron.rest.RestException: Unable to get column metadata
        at org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:95)
        at org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
        at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.metron.indexing.dao.search.InvalidSearchException: Unable to get column metadata
        at org.apache.metron.elasticsearch.dao.ElasticsearchSearchDao.buildSearchRequest(ElasticsearchSearchDao.java:155)
        at org.apache.metron.elasticsearch.dao.ElasticsearchSearchDao.search(ElasticsearchSearchDao.java:127)
        at org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:196)
        at org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertSearchDao.search(ElasticsearchMetaAlertSearchDao.java:81)
        at org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:209)
        at org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:92)
        ... 87 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        ... 1 more
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb0099 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb009b for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb009a for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:24 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:24 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb009a for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

[mmiklavc]

EDIT - that's the rest logs, not management UI logs. When I shutdown the management UI, the exceptions stop.

[mmiklavc]

Forgot to click submit on these Q's last week.

[mmiklavc]

Also, it's probably worth adding a simple explanation or diagram that talks about the message routing - e.g. I'm assuming we have these endpoints available via 2 distinct URL's when run locally?

[mmiklavc]

I will be looking into this further, but I see a lot of the following in the management UI logs. This repeated for more than 80k lines in a matter of minutes:

18/12/06 20:13:23 INFO zookeeper.ClientCnxn: Opening socket connection to server node1/192.168.66.121:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/06 20:13:23 WARN zookeeper.ClientCnxn: Session 0x1678019c6bb0099 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
18/12/06 20:13:23 ERROR controller.RestExceptionHandler: Encountered error: Unable to get column metadata
org.apache.metron.rest.RestException: Unable to get column metadata
        at org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:95)
        at org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
        at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at 
...

Turns out this was Zookeeper having gone down after HDFS filled up. Ambari had a lag in reporting. Independent of this PR, we might want to see about addressing REST api connectivity issues in the management UI. As it stands, the UI is left orphaned and inoperable without much explanation to the end user when something goes wrong when communicating with REST.

[mmiklavc]

@merrimanr Just summing up a few things I think we should have come out of this PR per our offline convo earlier:

1. Update README's with some explanation of the multiple UI and REST API communication architecture - e.g. I had expected there was a way to have the REST application and Angular application assets be served up by (or have the REST calls handled through) the same web container context, but that is not how this is setup. The pre-Knox existing architecture uses ExpressJS within the Angular app hosting container to do a very straightforward proxying of requests to the REST API as opposed to having a heavier weight facade (e.g. a Spring J2EE application) for an abstraction managing those requests to REST. This is fine, but now that we've added Knox to the mix, we should be very clear about what requests and responses go to which hosts/servers and why.
2. Developer documentation - simple diagram and/or write up that expresses what the request/response lifecycle looks like with and without Knox. This relates to number 1, but is more about showing the before/after of enabling Knox.
3. Explanation of the choice to NOT use Knox's URL rewriting capabilities. Per our convo earlier, the choice is between setting a base root that's used by Knox for routing requests (similar to the function of a controller in MVC architecture) to the appropriate host, or leaving the REST application alone and leveraging Knox's regular expression pattern matching to handle rewriting URL's that show up in the client browser.
4. EDIT - Fix logout bug
5. EDIT - Fix Travis Cypress test failure
6. I'm going to make a PR against your PR to add the Management UI to this.

Here's the basic request/response lifecycle as I understand it:

Static asset call for Angular application

This is for javascript, html, and images. I realize "static" may be a misnomer for Angular javascript, but it's static insofar as the web container is concerned:

Client request
-> knox server
-> angular web server

Server response
<- angular web server
<- knox server
<- client

Dynamic/REST content call.

Knox is now replacing ExpressJS entirely as the proxy mechanism. Note, ExpressJS didn't perform any function for static assets, only for calls to 3rd parties like the REST app. This is necessary because CORS restricts a browser from making the calls directly to REST (or any other non-origin host that's different from the original host that served up the client-side Angular app). This could theoretically be handled by a proper server side web application, but my understanding is that this would be superfluous given the API contract we've enacted with the REST API, i.e. it handles requests/responses in JSON that an AngularJS application can process easily.

Client request
-> knox server
-> rest api server

Server response
<- rest api server
<- knox server
<- client

[merrimanr]

@mmiklavc I've made several updates to the PR. I realized in my testing that for this feature to work correctly, the Knox SSO code should be included so I merged in #1281.

In addition to resolving those merge conflicts, I also added a unit test for the Knox SSO servlet filter and the documentation you requested. Let me know if the doc makes sense and has enough detail. I'm also debating copying the testing instructions from this PR into the README as Knox setup instructions. A follow on PR that adds this feature to the Mpack is going up soon so are manual instructions worth documenting? I also fixed a couple more UI bugs (login page redirection and the logout bug you mentioned) and tested @sardell's fix in #1301.

I think that covers points 1, 2, 4, and 5. For 3, I added an inline comment to index.html explaining the relative base path setting. Do you still think we need to add something to the README for this? For 6, I was going to start on it if you haven't already.

I've updated the PR description with end to end instructions for testing Knox SSO. I believe this is ready for testing and further review.

[mmiklavc]

A follow on PR that adds this feature to the Mpack is going up soon so are manual instructions worth documenting?

Not necessary. I think just link the mpack PR to this one in the main comments and mention "don't merge until" so that they are committed together.

[merrimanr]

The latest commit adds the Management UI as a Knox service. I've updated the PR description and testing instructions to include it. Setup is very similar to the Alerts UI. I am still working on getting the unit tests passing but it should function correctly.

Sounds good wrt to mpack vs setup documentation @mmiklavc. I will open another PR that references this PR for the mpack changes.

[merrimanr]

The latest commit addresses the metron-config unit test failures. This PR should be ready for review.

[mmiklavc]

This is looking pretty good @merrimanr. I have some questions and clarifications, but I think it's very close. A couple other things in addition to what I've commented inline:

1. Can you talk about the rewrite rules at all? What are they currently doing and when would someone need to modify those files?
2. Can you add any comment on the LDAP config with roles? Maybe a very brief cookie cutter example of how a user's LDAP users and groups will translate into Metron's interpretation of that info.

[merrimanr]

1. Can you talk about the rewrite rules at all? What are they currently doing and when would someone need to modify those files?

There is not much to say about this because there isn't any significant rewriting going on. Knox is simply forwarding the requests as is. This is the same for all 3 services (Alerts UI, Management UI, and REST) and you'll notice all rewrite.xml files are structured the same. The only requirement I had an issue with is that static assets needs to be prepended with the Knox path. This was solved by making the base href relative in index.html. I added a comment in the metron-alerts index.html file but missed it in metron-config. I will go back and add that comment. I cannot think of a scenario where changing a rewrite.xml file would be necessary.

2. Can you add any comment on the LDAP config with roles? Maybe a very brief cookie cutter example of how a user's LDAP users and groups will translate into Metron's interpretation of that info.

You're right, we should document how this works. I will add a section to the README.

[merrimanr]

@mmiklavc the latest commits should address the changes you requested. Let me know if I missed something. I ran this up in full dev again and everything still works.

[mmiklavc]

Latest changes look good @merrimanr. I'm nearly finished with my testing against #1308, which includes this PR. I'm +1 on this PR, which I presume will be merged at the same time as METRON-1945 once it gets +1'ed