运行一段时间后上报数据失败

[duanzonglong]

---

Bug

6.1.0
docker tomcat:7.0.77-jre8

• c-wms.com

• What happen?
  运行一段时间后 所有的节点都无法上报信息 必须重启agent所在服务 一下错误是我查找其中一个节点的消息
  DEBUG 2019-05-09 09:17:48:103 DubboServerHandler-10.42.188.146:20883-thread-62 TraceSegmentServiceClient : One trace segment has been abandoned, cause by buffer is full.
  DEBUG 2019-05-09 09:17:48:117 DubboServerHandler-10.42.188.146:20883-thread-62 TraceSegmentServiceClient : One trace segment has been abandoned, cause by buffer is full.
  ERROR 2019-05-09 09:17:48:123 DubboServerHandler-10.42.188.146:20883-thread-62 InstMethodsInterWithOverrideArgs : class[class com.rabbitmq.client.impl.ChannelN] before method[basicPublish] intercept failure
  java.lang.NullPointerException
  ERROR 2019-05-09 09:17:48:123 DubboServerHandler-10.42.188.146:20883-thread-62 InstMethodsInterWithOverrideArgs : class[class com.rabbitmq.client.impl.ChannelN] after method[basicPublish] intercept failure
  java.lang.IllegalStateException: No active span.
  at org.apache.skywalking.apm.agent.core.context.TracingContext.activeSpan(TracingContext.java:382)
  at org.apache.skywalking.apm.agent.core.context.ContextManager.activeSpan(ContextManager.java:161)
  at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:165)
  at org.apache.skywalking.apm.plugin.rabbitmq.RabbitMQProducerInterceptor.afterMethod(RabbitMQProducerInterceptor.java:94)
  at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInterWithOverrideArgs.intercept(InstMethodsInterWithOverrideArgs.java:104)
  at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java)
  at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:639)
  at sun.reflect.GeneratedMethodAccessor802.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:917)
  at com.sun.proxy.$Proxy196.basicPublish(Unknown Source)
  at org.springframework.amqp.rabbit.core.RabbitTemplate.doSend(RabbitTemplate.java:1510)
  at org.springframework.amqp.rabbit.core.RabbitTemplate$3.doInRabbit(RabbitTemplate.java:704)
  at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1436)
  at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1392)
  at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:700)
  at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:768)
  at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:764)

[wu-sheng]

DEBUG 2019-05-09 09:17:48:103 DubboServerHandler-10.42.188.146:20883-thread-62 TraceSegmentServiceClient : One trace segment has been abandoned, cause by buffer is full.

This is not a failure. This is a debug log, showing you, backend performance may be not enough to support. So, agent abandons the trace, to avoid to crash your application. This is the key fail-safe mechanism in the agent.

ERROR 2019-05-09 09:17:48:123 DubboServerHandler-10.42.188.146:20883-thread-62 InstMethodsInterWithOverrideArgs : class[class com.rabbitmq.client.impl.ChannelN] after method[basicPublish] intercept failure
java.lang.IllegalStateException: No active span.

This related to your rabbitmq plugin, maybe not compatible. You need to debug the plugin and find out why. Or provide demo codes to reproduce the error. THe demo codes should be easy for us to run at local.