[Bug] Incorrect trace from dubbo 3.x plugin in more complex scenario due to the incorrect Side judgement(consumer or provider)

[honganan]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

I think I have found a bug of the dubbo 3.x plugin due to incorrect using of RpcContext, and the test scenario we have now is too simple to find it:

If there is an invoking chain which is A -> B -> C, B is a Dubbo provider and also a consumer to C. Obviously B should report 2 span (a Entry Span from A and a Exit Span to C). But in fact it only reports one Span in the Segment, and it's content is chaotic.

As it used the RpcServiceContext to judge if now is in the consumer or provider side, if the above B service is both a provider and consumer, the RpcServiceContext is not changed when it calls C. Combined with Skywalking's createSpan logic, the result becomes puzzling.

After replaced to my new written judge method, the problem resolved.

What you expected to happen

Service B should report a segment contains an EntrySpan and an ExitSpan.

How to reproduce

Modify the test scenario to add one more invocation from GreetService, here paste my test scenario key code:
GreetServiceImpl.java

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

package org.apache.skywalking.apm.testcase.dubbo3.services.impl;

import org.apache.dubbo.config.ReferenceConfig;
import org.apache.dubbo.config.RegistryConfig;
import org.apache.skywalking.apm.testcase.dubbo3.services.ExceptionService;
import org.apache.skywalking.apm.testcase.dubbo3.services.GreetService;

public class GreetServiceImpl implements GreetService {

    private RegistryConfig registryConfig = new RegistryConfig("zookeeper://127.0.0.1:2181");

    public GreetServiceImpl() {
    }

    @Override
    public String doBusiness(String s) {
        ReferenceConfig<ExceptionService> referenceConfig = new ReferenceConfig<>();
        referenceConfig.setRegistry(registryConfig);
        referenceConfig.setInterface(ExceptionService.class);
        referenceConfig.setScope("remote");
        referenceConfig.setInjvm(false);
        referenceConfig.setTimeout(500000);
        referenceConfig.setAsync(false);
        referenceConfig.setCheck(false);

       ExceptionService exceptionService = referenceConfig.get();
        try {
            exceptionService.exceptionCall();
        } catch (Exception e) {
            // do nothing
            e.printStackTrace();
        }
        return "{name:'" + s + "'}";
    }
}

ExceptionService.java

package org.apache.skywalking.apm.testcase.dubbo3.services;

/**
 * @author honganan(weiyang.hong)
 * @date 2022/2/9 10:31 上午
 */
public interface ExceptionService {

    void exceptionCall();
}

ExceptionServiceImpl.java

package org.apache.skywalking.apm.testcase.dubbo3.services.impl;

import org.apache.skywalking.apm.testcase.dubbo3.services.ExceptionService;

/**
 * @author honganan(weiyang.hong)
 * @date 2022/2/9 10:31 上午
 */
public class ExceptionServiceImpl implements ExceptionService {
    @Override
    public void exceptionCall() {
        throw new RuntimeException("test exception!");
    }
}

Application.java

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

package org.apache.skywalking.apm.testcase.dubbo3;

import org.apache.dubbo.config.*;
import org.apache.dubbo.rpc.model.ApplicationModel;
import org.apache.skywalking.apm.testcase.dubbo3.services.ExceptionService;
import org.apache.skywalking.apm.testcase.dubbo3.services.GreetService;
import org.apache.skywalking.apm.testcase.dubbo3.services.impl.ExceptionServiceImpl;
import org.apache.skywalking.apm.testcase.dubbo3.services.impl.GreetServiceImpl;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        new EmbeddedZooKeeper(2181, false).start();
        SpringApplication.run(Application.class, args);
    }

    @Configuration
    public static class DubboConfiguration {

        private ApplicationConfig applicationConfig = new ApplicationConfig(Application.class.getSimpleName());

        private RegistryConfig registryConfig = new RegistryConfig("zookeeper://127.0.0.1:2181");

        private ProtocolConfig protocolConfig = new ProtocolConfig("dubbo", 20080);

        public DubboConfiguration() {
            ApplicationModel.getConfigManager().setApplication(applicationConfig);
        }

        @Bean(destroyMethod = "unexport")
        public ServiceConfig<GreetService> service() {
            ServiceConfig<GreetService> serviceConfig = new ServiceConfig<>();
            serviceConfig.setApplication(applicationConfig);
            serviceConfig.setRegistry(registryConfig);
            serviceConfig.setProtocol(protocolConfig);
            serviceConfig.setInterface(GreetService.class);
            serviceConfig.setRef(new GreetServiceImpl());
//            serviceConfig.setRef(new GreetServiceImpl());
            serviceConfig.setTimeout(500000);
            serviceConfig.export();
            return serviceConfig;
        }

        @Bean(destroyMethod = "unexport")
        public ServiceConfig<ExceptionService> service2() {
            ServiceConfig<ExceptionService> serviceConfig = new ServiceConfig<>();
            serviceConfig.setApplication(applicationConfig);
            serviceConfig.setRegistry(registryConfig);
            serviceConfig.setProtocol(protocolConfig);
            serviceConfig.setInterface(ExceptionService.class);
            serviceConfig.setRef(new ExceptionServiceImpl());
            serviceConfig.setTimeout(500000);
            serviceConfig.export();
            return serviceConfig;
        }

        @Bean(destroyMethod = "destroy")
        public ReferenceConfig<GreetService> reference() {
            ReferenceConfig<GreetService> referenceConfig = new ReferenceConfig<>();
            referenceConfig.setApplication(applicationConfig);
            referenceConfig.setRegistry(registryConfig);
            referenceConfig.setInterface(GreetService.class);
            referenceConfig.setScope("remote");
            referenceConfig.setInjvm(false);
            referenceConfig.setTimeout(500000);
            referenceConfig.setAsync(false);
            referenceConfig.setCheck(false);

            return referenceConfig;
        }

    }
}

Now the GreetService's segment reported likes blow:

- segmentId: 5698796dab2f4e1eb71c4e13130e61d7.77.16443991628440000
    errorSample: true
    entryUrl: /dubbo-3.x-scenario/case/dubbo
    spans:
    - operationName: org.apache.skywalking.apm.testcase.dubbo3.services.ExceptionService.exceptionCall()
      operationId: 0
      parentSpanId: -1
      spanId: 0
      spanLayer: RPCFramework
      startTime: 1644399162844
      endTime: 1644399162916
      componentId: 3
      isError: false
      spanType: Entry
      peer: 172.17.0.5:41862
      skipAnalysis: false
      tags:
      - {key: url, value: '....'}
      - {key: arguments, value: helloWorld}
      refs:
      - {parentEndpoint: /dubbo-3.x-scenario/case/dubbo, networkAddress: '172.17.0.5:0',
        refType: CrossProcess, parentSpanId: 1, parentTraceSegmentId: 5698796dab2f4e1eb71c4e13130e61d7.57.16443991624090000,
        parentServiceInstance: 26c09fc2fb2e4327ad53a957f82ea920@172.17.0.5, parentService: dubbo-3.x-scenario,
        traceId: 5698796dab2f4e1eb71c4e13130e61d7.57.16443991624090001}

It's expected like:

  - segmentId: 5698796dab2f4e1eb71c4e13130e61d7.77.16443991628440000
    errorSample: true
    entryUrl: /dubbo-3.x-scenario/case/dubbo
    spans:
    - operationName: org.apache.skywalking.apm.testcase.dubbo3.services.ExceptionService.exceptionCall()
      operationId: 0
      parentSpanId: 0
      spanId: 1
      spanLayer: RPCFramework
      startTime: 1644399162876
      endTime: 1644399162915
      componentId: 3
      isError: true
      spanType: Exit
      peer: 172.17.0.5:0
      skipAnalysis: false
      tags:
      - {key: url, value: 'dubbo://172.17.0.5:0/org.apache.skywalking.apm.testcase.dubbo3.services.ExceptionService.exceptionCall()'}
      logs:
      - logEvent:
        - {key: event, value: error}
        - {key: error.kind, value: java.lang.RuntimeException}
        - {key: message, value: test exception!}
        - key: stack
          value: |
            java.lang.RuntimeException: test exception!
            at org.apache.skywalking.apm.testcase.dubbo3.services.impl.ExceptionServiceImpl.exceptionCall(ExceptionServiceImpl.java:13)....
    - operationName: org.apache.skywalking.apm.testcase.dubbo3.services.GreetService.doBusiness(String)
      operationId: 0
      parentSpanId: -1
      spanId: 0
      spanLayer: RPCFramework
      startTime: 1644399162844
      endTime: 1644399162916
      componentId: 3
      isError: false
      spanType: Entry
      peer: 172.17.0.5:41862
      skipAnalysis: false
      tags:
      - {key: url, value: 'dubbo://172.17.0.5:20080/org.apache.skywalking.apm.testcase.dubbo3.services.GreetService.doBusiness(String)'}
      - {key: arguments, value: helloWorld}
      refs:
      - {parentEndpoint: /dubbo-3.x-scenario/case/dubbo, networkAddress: '172.17.0.5:0',
        refType: CrossProcess, parentSpanId: 1, parentTraceSegmentId: 5698796dab2f4e1eb71c4e13130e61d7.57.16443991624090000,
        parentServiceInstance: 26c09fc2fb2e4327ad53a957f82ea920@172.17.0.5, parentService: dubbo-3.x-scenario,
        traceId: 5698796dab2f4e1eb71c4e13130e61d7.57.16443991624090001}

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[honganan]

#7203

[wu-sheng]

@honganan Besides submitting a pull request to fix this, I am feeling you should submit this bug to Dubbo too. Because the wrong metadata in RpcServiceContext is definitely unexpected.

[honganan]

I will fix it these two days as the test scenario need to be improved, and Perhaps it can't be saying as a bug for Dubbo, it's just a RPC framework and normally users just do nothing with Context when call another service. We should attention this but.

[honganan]

I am not very familiar with Dubbo 3.0, and it separates RpcContext to several Classes. May be after I will learn more about it and report the issue if needed.

[wu-sheng]

        boolean isConsumer;
        URL url = serviceContext.getUrl();
        if (url == null) {
            url = serviceContext.getConsumerUrl();
            isConsumer = url != null;
        } else {
            isConsumer = serviceContext.isConsumerSide();
        }

Which condition leads to the failure of the isConsumer value? RPC context should be isolated. Let me ask to Dubbo community.

[honganan]

The else branch:

isConsumer = serviceContext.isConsumerSide();

at this time the serviceContext is still the provider side's, it seems understandable as the provider is not finished yet.

[wu-sheng]

Let me ask, this seems a bug from my understanding.

[honganan]

I see it, Let's see how did they designed the Contexts.

[wu-sheng]

@naah69 It would be better you could involve in this.