Skip to content
This repository was archived by the owner on Aug 20, 2025. It is now read-only.

Conversation

@nickwallen
Copy link
Contributor

@nickwallen nickwallen commented Sep 11, 2018

Right now, the user can only configure Metron's client JAAS in limited ways. They can configure the principal name and key tab path, but few other things. For example, I cannot easily turn on debugging today. If I make a change to $METRON_HOME/client_jaas.conf to set debug=true, Ambari will overwrite my change when any of the topologies are restarted.

With this PR the user can now edit a template in Ambari that is used to generate $METRON_HOME/client_jaas.conf. This allows user to turn on debug logs, change the service name, or add additional options required by their environment.

screen shot 2018-09-11 at 2 12 35 pm

Testing

  1. Stand-up a development environment.

  2. Ensure alerts are visible in the Alerts UI.

  3. Kerberize that development environment.

  4. Ensure alerts are visible in the Alerts UI.

  5. Ensure that Metron's client_jaas.conf was created correctly.

    [root@node1 0.6.0]# source /etc/default/metron
    [root@node1 0.6.0]# cat $METRON_HOME/client_jaas.conf
    
    StormClient {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=false
       storeKey=false
       useTicketCache=true
       serviceName="nimbus"
       principal="metron@EXAMPLE.COM";
    };
    Client {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/etc/security/keytabs/metron.headless.keytab"
       storeKey=true
       useTicketCache=false
       serviceName="zookeeper"
       principal="metron@EXAMPLE.COM";
    };
    KafkaClient {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       keyTab="/etc/security/keytabs/metron.headless.keytab"
       storeKey=true
       useTicketCache=false
       serviceName="kafka"
       principal="metron@EXAMPLE.COM";
    
  6. In Ambari, go to Metron > Configs > Advanced > Advanced metron-client-jaas-conf and alter the template in some way. For example, you could add debug=true to each stanza.

  7. Restart the Enrichment topology.

  8. Validate that Metron's client_jaas.conf was updated based on the changes you made in Ambari.

    [root@node1 0.6.0]# cat client_jaas.conf
    StormClient {
       com.sun.security.auth.module.Krb5LoginModule required
       debug=true
       useKeyTab=false
       storeKey=false
       useTicketCache=true
       serviceName="nimbus"
       principal="metron@EXAMPLE.COM";
    };
    Client {
       com.sun.security.auth.module.Krb5LoginModule required
       debug=true
       useKeyTab=true
       keyTab="/etc/security/keytabs/metron.headless.keytab"
       storeKey=true
       useTicketCache=false
       serviceName="zookeeper"
       principal="metron@EXAMPLE.COM";
    };
    KafkaClient {
       com.sun.security.auth.module.Krb5LoginModule required
       debug=true
       useKeyTab=true
       keyTab="/etc/security/keytabs/metron.headless.keytab"
       storeKey=true
       useTicketCache=false
       serviceName="kafka"
       principal="metron@EXAMPLE.COM";
    

Pull Request Checklist

  • Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
  • Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
  • Has your PR been rebased against the latest commit within the target branch (typically master)?
  • Have you included steps to reproduce the behavior or problem that is being changed or addressed?
  • Have you included steps or a guide to how the change may be verified and tested manually?
  • Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
  • Have you written or updated unit tests and or integration tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

@nickwallen
Copy link
Contributor Author

CI failed due to unrelated transient test failure...

SensorEnrichmentConfigControllerIntegrationTest.test:231 Status expected:<404> but was:<200>

@nickwallen nickwallen closed this Sep 11, 2018
@nickwallen nickwallen reopened this Sep 11, 2018
@nickwallen
Copy link
Contributor Author

CI failed due to unrelated transient test failure...

Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 289.241 sec <<< FAILURE! - in org.apache.metron.profiler.integration.ProfilerIntegrationTest
testProcessingTimeWithTimeToLiveFlush(org.apache.metron.profiler.integration.ProfilerIntegrationTest)  Time elapsed: 135.735 sec  <<< FAILURE!
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertTrue(Assert.java:52)
	at org.apache.metron.profiler.integration.ProfilerIntegrationTest.testProcessingTimeWithTimeToLiveFlush(ProfilerIntegrationTest.java:210)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

I am going to take a look at this failure (outside of this PR)

@nickwallen nickwallen closed this Sep 12, 2018
@nickwallen nickwallen reopened this Sep 12, 2018
@mmiklavc
Copy link
Contributor

Took a look over the source and looks good to me there. I do want to spin this up in full dev before I give it my +1. One question regarding the change from j2/Jinja templates. Just curious, was that necessary to make the jaas config available for user modification in a text field in Ambari or for other reasons?

@nickwallen
Copy link
Contributor Author

@mmiklavc: Just curious, was that necessary to make the jaas config available for user modification in a text field in Ambari or for other reasons?

Yes, that is the only way I've seen it done. Its similar in the other Mpacks, like Kafka, etc.

@mmiklavc
Copy link
Contributor

Hey @nickwallen, I see a number of exceptions in the metron-rest.log. It dies shortly thereafter.

18/09/18 00:51:41 WARN context.AnnotationConfigServletWebServerApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaConfig': Unsatisfied dependency expressed through field 'zkClient'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zkClient' defined in class path resource [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
18/09/18 00:51:41 ERROR imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up
java.lang.IllegalStateException: instance must be started before calling this method
        at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:363)
        at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.doRefreshData(TreeCache.java:244)
        at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.refresh(TreeCache.java:209)
        at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.wasCreated(TreeCache.java:263)
        at org.apache.curator.framework.recipes.cache.TreeCache$TreeNode.processResult(TreeCache.java:397)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:728)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:505)
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
18/09/18 00:51:41 INFO zookeeper.ZooKeeper: Session: 0x165e9a7dc4a002a closed
18/09/18 00:51:41 INFO jpa.LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default'
18/09/18 00:51:41 INFO hikari.HikariDataSource: HikariPool-1 - Shutdown initiated...
18/09/18 00:51:41 INFO zookeeper.ClientCnxn: EventThread shut down
18/09/18 00:51:41 INFO hikari.HikariDataSource: HikariPool-1 - Shutdown completed.
Sep 18, 2018 12:51:41 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Tomcat]
18/09/18 00:51:41 INFO logging.ConditionEvaluationReportLoggingListener:

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
18/09/18 00:51:41 ERROR boot.SpringApplication: Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kafkaConfig': Unsatisfied dependency expressed through field 'zkClient'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zkClient' defined in class path resource [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:587)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:91)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:373)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:578)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
        at org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zkClient' defined in class path resource [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1065)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:584)
        ... 19 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' threw exception; nested exception is java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
        ... 31 more
Caused by: java.lang.IllegalArgumentException: File /usr/metron/0.6.0/client_jaas.confcannot be read.
        at org.I0Itec.zkclient.ZkClient.isZkSaslEnabled(ZkClient.java:904)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
        at org.apache.metron.rest.config.ZookeeperConfig.zkClient(ZookeeperConfig.java:63)
        at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$e6070ab6.CGLIB$zkClient$1(<generated>)
        at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$e6070ab6$$FastClassBySpringCGLIB$$8bbe133d.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
        at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$e6070ab6.zkClient(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        ... 32 more

@mmiklavc
Copy link
Contributor

I think I see the problem. Is this meant to be only tied to a specific topology(ies) restarting to get deployed? There were some errors during service restart after Kerberos deployed which required me to restart a number of services manually. Since I'm running in full dev I only restarted a few necessary services, including REST and the UIs. Neither seems to have forced the jaas file to deploy.

@mmiklavc
Copy link
Contributor

I restarted indexing and the jaas shows up. I think this may just be missing a couple extra service dependencies, but it's close.

@nickwallen
Copy link
Contributor Author

nickwallen commented Sep 18, 2018

@mmiklavc Nothing changed about how the JAAS template gets deployed (like which service triggers the deployment).

The template gets deployed when any service calls storm_security_setup, which includes Parser, Enrichment, Profiler, Indexing, and the mysterious Metron Client.

The UIs and REST never triggered a deployment of the JAAS template. I assume because they don't directly use it.

<property>
<name>content</name>
<display-name>metron_client_jaas template</display-name>
<description>Metron client JAAS configuration</description>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The template that previously lived in client_jaas.conf.j2 is now here. This allows the user to edit the template in Ambari.


File(ambari_format('{client_jaas_path}'),
content=Template('client_jaas.conf.j2'),
content=InlineTemplate(params.metron_client_jaas_conf_template),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of populating the template from the static .j2 file, the template now comes from the configuration. In MPack parlance, its now using an InlineTemplate.

@mmiklavc
Copy link
Contributor

It sounds like this may be an existing bug. That stacktrace makes it pretty clear that the REST api wants the jaas file for configuring the zkClient.

@nickwallen
Copy link
Contributor Author

Right. So probably only shows up because kerberization in Full Dev never comes up cleanly (mainly because of the Elasticsearch service check always fails; ES is 'yellow' not 'green').

I could add the storm_security_setup to the REST service. Can we tackle that as a separate PR? I'd want to refactor the code just a bit to make it clear that it is not necessarily just a "Storm" security setup, if in fact REST relies on it.

@mmiklavc
Copy link
Contributor

@nickwallen - sounds great to me. Can you create a Jira to track that and link it to this PR? +1

@nickwallen
Copy link
Contributor Author

@asfgit asfgit closed this in b84c874 Sep 18, 2018
MohanDV pushed a commit to MohanDV/metron that referenced this pull request Sep 19, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants