Permission Denied when running Rwalk scripts

[Manno15]

With the recent changes in accumulo #1828, some of the modules for RWalk now throw permission exceptions.

The specific one I ran into while running ./bin./rwalk All.xml and specifically./bin/rwalk Security.xml is below:

ThriftSecurityException(user:system_flash_superheroes_local, code:PERMISSION_DENIED)
	at org.apache.accumulo.server.security.SecurityOperation.authenticateUser(SecurityOperation.java:238)
	at org.apache.accumulo.server.client.ClientServiceHandler.authenticateUser(ClientServiceHandler.java:150)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$6(TraceUtil.java:235)
	at com.sun.proxy.$Proxy38.authenticateUser(Unknown Source)
	at org.apache.accumulo.core.clientImpl.thrift.ClientService$Processor$authenticateUser.getResult(ClientService.java:2608)
	at org.apache.accumulo.core.clientImpl.thrift.ClientService$Processor$authenticateUser.getResult(ClientService.java:2587)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
	at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:114)
	at org.apache.thrift.server.Invocation.run(Invocation.java:18)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

There are also several warnings thrown now for each permission in our auditing. Some examples below:

operation:  failed; user: system_flash_superheroes_local; 
action:  changeAuthorizations; targetUser: system_flash_superheroes_local;  
authorizations:  Towels,Paper,Brush,Asparagus,Fishsticks,PotatoSkins,Ribs,Celery;  
exception: ThriftSecurityException(user:system_flash_superheroes_local,  
code:PERMISSION

operation: failed; user: root; checking permission DROP_USER on table_flash_superheroes_local denied; 
exception: ThriftSecurityException(user:table_flash_superheroes_local, 
code:USER_DOESNT_EXIST)

operation:  failed; user: table_flash_superheroes_local; 
action:  revokeTablePermission;
permission: BULK_IMPORT; targetTable:  security_flash_superheroes_local; targetUser:  system_flash_superheroes_local;; 
exception:  ThriftSecurityException(user:table_flash_superheroes_local,  
code:PERMISSION_DENIED)

This seems to happen for each permission type, either with Permission_Denied or for User_Doesn't_Exist.
New one below:

ERROR Framework Error during random walk
 java.lang.Exception: Error running node Security.xml
	at org.apache.accumulo.testing.randomwalk.Module.visit(Module.java:370)
	at org.apache.accumulo.testing.randomwalk.Framework.run(Framework.java:48)
	at org.apache.accumulo.testing.randomwalk.Framework.main(Framework.java:92)
Caused by: org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_CREDENTIALS for user system_flash_superheroes_local - Username or Password is Invalid

[Manno15]

I am still investigating the specific modules that throw this error.

[DomGarguilo]

I was able to reproduce this error, however every time I run rwalk the java.lang.Exception displays a different node.
i.e.

ERROR Framework Error during random walk
 java.lang.Exception: Error running node Security.xml

ERROR Framework Error during random walk
 java.lang.Exception: Error running node Conditional.xml

Aside from these portions of the error message the rest seems to be the same.
In addition, when running ./bin./rwalk Image.xml I get:

ERROR Framework Error during random walk
 org.apache.accumulo.core.client.AccumuloException: java.lang.NullPointerException

[Manno15]

Security.xml seems to be the main one that throws these errors/warnings which makes sense. I still saw some in Concurrent.xml as well.

[milleruntime]

I think there were some concurrency issues with Security module so the last few times I ran RW, I just removed it from the config. All of the other ones should be working though.

[AlbertWhitlock]

When I run Image.xml, it works fine. I am not seeing the NullPointerException Dominic saw.

In fact, each one individually seems to run fine except Security.xml. I can replicate the original error: the PERMISSION_DENIED. Earlier in the run I am seeing an error in 'security.TableOp' node when trying to write. Currently I am trying to determine where/when/how these permissions get set.

[Manno15]

After looking at the debug logs of Securit.xml closely, everything seems to be running as expected. Some notes below:

There is an expected MutationsRejectedException that gets thrown in the TableOps Write case which accumulo will still log no matter what. That has been apart of the confusion.

GetSummaries near the bottom of TableOps isn't supported and rwalk will throw an exception near the end of its run because of it. Not quite sure if that is causing any issues or not but commenting that line out will remove the exception in the logs.

The biggest issue is the overload of logs in the accumulo monitor. Both warnings and errors. All of them being ThriftSecurityExceptions. After reviewing everything, this could just be the byproduct of the test and isn't actually causing any issues. If that is the case then the best course of action would be to either suppress the logs or just ignore them when running rwalk.

[ctubbsii]

If that is the case then the best course of action would be to either suppress the logs or just ignore them when running rwalk.

Suppressing warnings that are expected is better than ignoring them... otherwise, people will continue to be confused.

[ctubbsii]

Since it looks like this isn't really a problem, but expected behavior when the test verifies that a user in unable to perform an unauthorized action. So, I'll close this. Thanks for investigating, @Manno15 and @AlbertWhitlock