Skip to content

Conversation

@lokeshj1703
Copy link
Contributor

What changes were proposed in this pull request?

After running decommissioning tests and aborting few decommissioning command, seeing PipelineNotFoundException INFO logs continuously in scm log.
Although these are INFO logs, raising the issue as the exceptions are thrown in loop.

2021-05-16 20:06:53,421 INFO org.apache.hadoop.ipc.Server: IPC Server handler 53 on 9860, call Call#102 Retry#23 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
	at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
2021-05-16 20:06:55,426 INFO org.apache.hadoop.ipc.Server: IPC Server handler 84 on 9860, call Call#102 Retry#24 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
	at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
	at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
	at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
	at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
	at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
	at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
Options

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5248

How was this patch tested?

Adds UT

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lokeshj1703
Copy link
Contributor Author

@bharatviswa504 @bshashikant As per discussion, I have updated the PR.

Some APIs throw IOException directly. Those APIs will still need to be handled separately.

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM.

Can we open a Jira to handle IOException related.

*/
public PipelineNotFoundException() {
super();
super(ResultCodes.PIPELINE_NOT_FOUND);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this ResultCode is lost when received at client.
But I see the message with whole stacktrace with debug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest commit adds the result code in the protobuf. Did you see that behavior in latest commit @bharatviswa504 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I believe the ResultCodes in proto will help when we want to return ResultCodes in Response and construct the Exception based on that.

I believe when Rpc Layer constructs this Exception it used IOException default constructor and the ResultCode will be lost, and the message will be present.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RemoteException unwrapException code and instantiateException, only message will be saved.

    try {
      Class<?> realClass = Class.forName(getClassName());
      return instantiateException(realClass.asSubclass(IOException.class));
    } catch(Exception e) {
      // cannot instantiate the original exception, just return this
    }
    return this;
  }

private IOException instantiateException(Class<? extends IOException> cls)
      throws Exception {
    Constructor<? extends IOException> cn = cls.getConstructor(String.class);
    cn.setAccessible(true);
    IOException ex = cn.newInstance(this.getMessage());
    ex.initCause(this);
    return ex;
  }

@lokeshj1703
Copy link
Contributor Author

HDDS-5259 has been created to handle the IOException being thrown by the apis in StorageContainerLocationProtocol.

@lokeshj1703
Copy link
Contributor Author

@bharatviswa504 Thanks for the review! I have committed the PR to master branch.

@lokeshj1703 lokeshj1703 deleted the HDDS-5248 branch May 24, 2021 06:03
bharatviswa504 pushed a commit to bharatviswa504/hadoop-ozone that referenced this pull request Jul 25, 2021
…og. (apache#2267)

(cherry picked from commit 9080fc3)
Change-Id: I30245c1c2742676aad00776f81a328f3d75c7f43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants