Provide Solr-7.x-plugin by dmsolr · Pull Request #2853 · apache/skywalking

dmsolr · 2019-06-11T08:59:40Z

Please answer these questions before submitting pull request

Why submit this pull request?
Bug fix
New feature provided
Improve performance
Related issues
[Agent] Provide plugin for Solr-7.x #2501

wu-sheng

I suggest move this to optional plugin and add a document for this. Because this plugin should only work when agent installed in ElasticSearch server side, right?

dmsolr · 2019-06-11T09:07:42Z

The situation on the server side is more complicated than SolrJ.
I need some advice how to name operationName and other.

It will put the trace id into Solr MDC. Then Solr logs it in SLOW logger or where you want to by Log4j configuration file.

dmsolr · 2019-06-11T09:10:33Z

@wu-sheng yes, it works in Solr Server, likes ElasticSearch server-side.

wu-sheng · 2019-06-11T09:39:59Z

    * [transport-client](https://github.com/elastic/elasticsearch/tree/master/client/transport) 5.2.x-5.6.x
-  * [SolrJ](https://lucene.apache.org/solr) 7.0.0-7.7.1
+  * [Solr](https://github.com/apache/lucene-solr) 7.0.0-7.7.1
+    * [SolrJ](https://github.com/apache/lucene-solr/tree/master/solr/solrj) 7.0.0-7.7.1


Add Solr Server and versions as a subtype here.

wu-sheng · 2019-06-11T09:40:07Z

  * [Elasticsearch](https://github.com/elastic/elasticsearch)
    * [transport-client](https://github.com/elastic/elasticsearch/tree/master/client/transport) 5.2.x-5.6.x
-  * [SolrJ](https://lucene.apache.org/solr) 7.0.0-7.7.1
+  * [Solr](https://github.com/apache/lucene-solr) 7.0.0-7.7.1


Remove the version.

wu-sheng · 2019-06-11T09:42:29Z

Could you explain more about how many spans created for each request? And what are they?

coveralls · 2019-06-11T09:56:35Z

Coverage decreased (-0.004%) to 17.242% when pulling da99c25 on dmsolr:solr-7.x-plugin into ec731b2 on apache:master.

dmsolr · 2019-06-11T12:24:31Z

Here is a screenshot of solr collection. Obviously, it consists of 2 shards with 2 replicas per shard.

In query, we omit the replicas. One shard become a coordinator, when it receives a query from user. Because it is has a part of data in collection, It launch a request and send to others shards. Then coordinator have to merge the result that comes from all shards. In real scenario, it has serveral stages in usually.

It seems like above picture.

In update, the documents need to be routed and send to corresponding shard. Then shard will copy it to they replicas.

dmsolr · 2019-06-11T12:26:13Z

Solr is a distributed database.

dmsolr · 2019-06-11T12:29:42Z

@kezhenxu94， Do you have anything else for the explaination？

wu-sheng · 2019-06-11T13:02:20Z

OK. I think it is clear enough for the Solr server. Then let's talk about how this goes in topology and trace.

Does topology show Solr server cluster as a single one nodes? Like other distributed system.
Do you plan to show RPC between Solr server nodes?

I think the decision is based on the answers of the above questions.

dmsolr · 2019-06-11T17:12:45Z

@wu-sheng , Sorry, I have a mistake that I configured the agent.service_name in smae name.
In fact, they work in cluster.
By the way, Solr still uses http. So I define the SpanLayer as http.

wu-sheng · 2019-06-12T01:39:57Z

@TinyAllen I noticed in this case, there are instances of one service calling each others, the topology shows a little strange, please take a look.

wu-sheng · 2019-06-12T01:51:23Z

@dmsolr If the client SolrJ is instrumented, do the topology and metrics still make sense? I notice you are identifying the different shard as different service? Then which peer is SolrJ using?

Such as Solr server nodes are 10.5.1.23:1000(node1), 10.5.1.53:1000(node2). What is SolrJ plugin peer? I am assuming something like 10.5.1.23:1000,10.5.1.53:1000? If so, the topology can't show perfectly, because backend can't tell 10.5.1.23:1000,10.5.1.53:1000 is alias name of node1 or node2.

kezhenxu94 · 2019-06-12T02:13:28Z

Then which peer is SolrJ using?

Such as Solr server nodes are 10.5.1.23:1000(node1), 10.5.1.53:1000(node2). What is SolrJ plugin peer?

@wu-sheng The peer of SolrJ Plugin may not be always the same since there're something like 'load balancing' in the request. The SolrJ would contact the zookeeper first to get a Solr node and then send the request to that node (afterwards the request may require communication between Solr nodes such as aggregation), the request may not always be routed to the same node. Am I right? @dmsolr

dmsolr · 2019-06-12T02:48:29Z

Great, @kezhenxu94 .
Communication between nodes use SolrJ withou SolrJ-plugin.
You mean there is a problem when SolrJ-plugin works with Solr-plugin in one node, right. @wu-sheng

IanCao · 2019-06-12T03:07:05Z

+    @Override
+    public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
+                              Object ret) throws Throwable {
+        if (method.getName().equals("finishStage")) {


i'm a bit confused about if (method.getName().equals("distributedProcess")) in before method and if (method.getName().equals("finishStage")) { in after method

this two method name should be same.but this logic judgment is not

This span start at distributedProcess() and stop at finishStage().
But I'm pretty sure they always come in pairs.

IanCao · 2019-06-12T03:08:56Z

+
+    @Override
+    public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes, Throwable t) {
+        if (ContextManager.isActive()) {


!request.getPath().startsWith(SKIP_ADMIN_PREF) should be here

Yes, I notice that. Maybe it is not required. In usually, there is no active span which matches this condition here. (Unless other interceptors create.)

That there is a active span in context, not matter of who created, logs the exception when it is in the method with catching exception. I think.

wu-sheng · 2019-06-12T08:29:06Z

You mean there is a problem when SolrJ-plugin works with Solr-plugin in one node, right.

Could you test about SolrJ plugin works with one node Solr server and multiple nodes cluster? Especially for Topology. I highly doubt the topology will make sense if you do so when APP with Solr cluster.

dmsolr · 2019-06-12T13:06:41Z

I have tried it seems like ok.

wu-sheng · 2019-06-13T12:25:02Z

What is the user accessing all nodes? Maybe because you are using browser access Solr server?

dmsolr · 2019-06-13T13:18:12Z

Yes, I accessed all the nodes through the browser.
Accessing Solr via browser only connects to one node.
But SolrJ can access all nodes in a request, expecially writting data. This is because the node to which the document should be written is usually known. If partition rule is given.

wu-sheng · 2019-06-13T14:30:09Z

OK, if no peer looks like ip1:port1,ip2:port2... in SolrJ plugin, I think that is fine.

wu-sheng · 2019-06-14T00:44:03Z

cc #2871

wu-sheng · 2019-06-14T05:27:59Z

SolrJ plugin has been reverted, have to, because it breaks the agent.

asfgit · 2019-06-23T08:57:17Z

FAILURE

asfgit · 2019-06-23T08:57:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/skywalking-e2e/11/

dmsolr · 2019-07-04T06:59:28Z

Cases List

Case description Status Cases

solr 12 passed. 0 failed click me

wu-sheng · 2019-07-04T07:01:53Z

You need to fix conflict first.

dmsolr · 2019-07-04T07:11:39Z

The PR has delayed for long time, so I works in a new branch. This branch lose some commits. Could I close this and open another PR?

dmsolr · 2019-07-04T07:25:11Z

move to #3002

daming added 2 commits June 11, 2019 15:12

Prepare to PR

04e9c16

adjust operation name.

da99c25

wu-sheng reviewed Jun 11, 2019

View reviewed changes

wu-sheng added agent Language agent related. feature New feature plugin Plugin for agent or collector. Be used to extend the capabilities of default implementor. labels Jun 11, 2019

wu-sheng requested a review from IanCao June 11, 2019 09:40

wu-sheng added the TBD To be decided later, need more discussion or input. label Jun 11, 2019

IanCao reviewed Jun 12, 2019

View reviewed changes

wu-sheng added 2 commits June 12, 2019 21:14

Merge branch 'master' into solr-7.x-plugin

52d0512

Merge branch 'master' into solr-7.x-plugin

36391e0

kezhenxu94 mentioned this pull request Jun 17, 2019

Restore solrj-plugin #2888

Merged

3 tasks

Merge branch 'master' into solr-7.x-plugin

ec353b8

dmsolr closed this Jul 4, 2019

dmsolr deleted the solr-7.x-plugin branch July 10, 2019 11:54

Conversation

dmsolr commented Jun 11, 2019

Uh oh!

wu-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

dmsolr commented Jun 11, 2019

Uh oh!

dmsolr commented Jun 11, 2019

Uh oh!

wu-sheng Jun 11, 2019

Choose a reason for hiding this comment

Uh oh!

wu-sheng Jun 11, 2019

Choose a reason for hiding this comment

Uh oh!

wu-sheng commented Jun 11, 2019

Uh oh!

coveralls commented Jun 11, 2019

Uh oh!

dmsolr commented Jun 11, 2019

Uh oh!

dmsolr commented Jun 11, 2019

Uh oh!

dmsolr commented Jun 11, 2019

Uh oh!

wu-sheng commented Jun 11, 2019

Uh oh!

dmsolr commented Jun 11, 2019

Uh oh!

wu-sheng commented Jun 12, 2019

Uh oh!

wu-sheng commented Jun 12, 2019

Uh oh!

kezhenxu94 commented Jun 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmsolr commented Jun 12, 2019

Uh oh!

IanCao Jun 12, 2019

Choose a reason for hiding this comment

Uh oh!

dmsolr Jun 12, 2019

Choose a reason for hiding this comment

Uh oh!

IanCao Jun 12, 2019

Choose a reason for hiding this comment

Uh oh!

dmsolr Jun 12, 2019

Choose a reason for hiding this comment

Uh oh!

wu-sheng commented Jun 12, 2019

Uh oh!

dmsolr commented Jun 12, 2019

Uh oh!

wu-sheng commented Jun 13, 2019

Uh oh!

dmsolr commented Jun 13, 2019

Uh oh!

wu-sheng commented Jun 13, 2019

Uh oh!

wu-sheng commented Jun 14, 2019

Uh oh!

wu-sheng commented Jun 14, 2019

Uh oh!

asfgit commented Jun 23, 2019

Uh oh!

asfgit commented Jun 23, 2019

Uh oh!

dmsolr commented Jul 4, 2019

Cases List

Uh oh!

wu-sheng commented Jul 4, 2019

Uh oh!

dmsolr commented Jul 4, 2019

Uh oh!

dmsolr commented Jul 4, 2019

Uh oh!

Reviewers

Assignees

kezhenxu94 commented Jun 12, 2019 •

edited

Loading