Make 8.0.0 Core available. New protocol and register removed. by wu-sheng · Pull Request #4599 · apache/skywalking

wu-sheng · 2020-03-31T15:35:27Z

This would be a huge pull request, relating to protocol, agent and backend. 8.x would be an new core without back forward compatibility.
After the experiences of removing endpoint_inventory, I found out this
strategy is successful.
Especially, we totally get rid of register, so I want to do more.

SkyWalking 8.0.0
First, it is already unexpected for me, we have to move to 8.0.0 so
quickly, but after the discussion with @hanahmily , and thinking about this for several days, I think
we have to.

The key chances are following

Remove service, service instance, and network address register. The old
register protocols are totally going to be removed.
The agent doesn't need to do register anymore. Service name and Service
Instance name are generated by the agent itself, but the extra information,
such as IP, hostname, language, should report to backend separately.
Service Traffic should be added just like the endpoint traffic but keep
the time bucket as we need accurate service name in the given duration
Service Instance Traffic should be added too, with external information,
such as language, hostname.
Trace context propagation context should be changed to accept string in
service instance name, endpoint name and network address. This could ease
the agent logic, but also, requires changes in all language agent and
plugin test tool,
Trace report protocol requires to change too, in order to adopt the
string.
e2e tests have to ignore PHP and LUA at first, and remove the 6.x
compatibility test(doesn't support anymore).

The benefits we will get are

Don't worry about the inventory(s) that has been deleted randomly by end
users. (We received a lot of issue reports about this)
The upgrade could be easier erasing the whole storage and reboot the new
one. (Users don't feel comfortable about upgrade)
No hot-reboot case in the agent side
No cache of network address register information in the agent.
No service and service instance cache in the OAP
No register lock in the OAP
No file buffer mechanism in the OAP too, same as no register happens.

In my mind, I think this totally break upgrade is super meaningful and will
be good change. Even we break many things, they are easy to follow.
@mrproliu I think by following this, we need
to change the collaboration header to sw8 :) As no 7.1.0 release will
happen.

Link to mail list, https://lists.apache.org/thread.html/rda36fa8d191fc5750fc793993c69e14a917138bf97f9c2a461b72811%40%3Cdev.skywalking.apache.org%3E

…ng on the test case fix.

…tanceProperties as a new source. 3. Remove Month as downsamping and query step.

wu-sheng · 2020-04-01T03:03:53Z

The codes level key changes are

Set up traffic for service and service instance. Service Inventory and Instance Inventory are removed.
Add ServiceInstanceProperties as a new source.
Remove Month as downsamping and query step.
InfluxDB implementation doesn't depend on H2/MySQL, there is no metadb concept anymore. @dmsolr . I just made InfluxDB compiling passed, you need to add metadata query implementation.
Service Instance Update source is created for properties and heartbeat update.

…just the receiver, starting with trace analysis.

…8-core

wu-sheng · 2020-04-01T23:34:24Z

@kezhenxu94 @ascrutae @arugal @heyanlong @ElderJames

New protocol

…t them later.

wu-sheng · 2020-04-04T10:05:00Z

@apache/skywalking-committers , I just made the project package passed. Now we could move on the core tests.

I will continue working on local testing including H2, MySQL and ElasticSearch 6.
@dmsolr Please help on agent plugin tests and InfluxDB storage implementation.
@arugal Please help on go2sky project
@heyanlong Please help on PHP project
@ElderJames Please help on .net project
@ascrutae Please help on nodejs project.
@kezhenxu94 Please help on testing and new e2e setup
@mrproliu Please help on backend testing, especially profiling and LUA agent test.
@JaredTan95 Please help on backend testing.
@hanahmily Please run tests on mesh case after the above are done.
@TinyAllen @Fine0830 Please remove the month step support when UI time selector choose more than one month. In this case, still set to days, and Max days should be 60 days. The month step has been removed from the GraphQL.

2. Add MeshDataMock.

kezhenxu94 · 2020-04-09T11:04:30Z

Mark it as "Ready for review" as the test cases should be fixed now, and move to next step if anyone will review the code

wu-sheng · 2020-04-09T12:07:21Z

            exitSpan = parentSpan;
        } else {
            final int parentSpanId = parentSpan == null ? -1 : parentSpan.getSpanId();
-            if (StringUtil.isEmpty(remotePeer)) {


This deletion actually is not related to this PR. The lazy peer initialization is another feature :) @kezhenxu94

wu-sheng · 2020-04-09T12:24:44Z

Summary

This pull request is open for 10 days, and I worked on this 2 weeks ago. Thanks everyone to help making this PR and tests work now.

Here are the list of changes we made in this pull request. Top 3 are the principle changes, others are following these principles.

New agent and mesh report protocol.
New agent header protocol.
Service register, instance register and network address register have been removed permanently.
Service traffic, instance traffic and network alias metrics are added to replace the service, instance and network address inventory.
Register process has been removed.
Metrics stream process supports insert only mode, especially for traffic entities.
Metrics stream process supports no-downsampling mode for traffic entities and network alias.
Remove all register mechanism and cache in the java agent.
Remove MONTH step in GraphQL query.
Update UI to remove MONTH step query, the max query range is 60 days now.
Simplify the TTL to metrics and record. And the unit has been formatted in Day unit. No specific TTL for ElasticSearch storage.
Buffer mechanism of trace receiver and mesh receiver has been removed due to no register.
New service id, instance id and endpoint id rules, including service relation, instance relation and endpoint relation id rules.
Java agent support keep tracing mode, meaning, agent generating tracing context even the backend is unconnected/unavailable.
Plugin test tool up to date, in order to support new protocol.
Plugin tests expected data files updated.
E2E tests updated.
Telemetry of Grafana config has been merged into one.
Documentation updates.
[TBD] InfluxDB storage implementation is not available, need @dmsolr to fix later, in order to reduce the master change block by this PR.

FYI @apache/skywalking-committers

mrproliu

I have a test with the profile and alarm module, it works well. And test Nginx agent on the local environment, it works well too.

dmsolr

I have tested java agent and agent plugin. It looks good to me.

wu-sheng · 2020-04-10T00:16:40Z

@kezhenxu94 Are you going to do code level review?

kezhenxu94 · 2020-04-10T00:26:33Z

@kezhenxu94 Are you going to do code level review?

I can do another round of review recently, but that’s not a blocker to merge this, it looks good to me generally, just merge it if necessary

wu-sheng · 2020-04-10T00:36:42Z

@kezhenxu94 I am adjusting some documentations, after that, I will merge this.

hanahmily

LGTM

heyanlong

PHP test success

rainbend

go2sky test success

wu-sheng added 3 commits March 31, 2020 17:50

1. Update protocol. 2.Finish most changes at the agent side. 3. Worki…

212339c

…ng on the test case fix.

Clean the test issue. Agent and plugins are ready.

1322075

Add IDManager for further refactor.

1e4a66c

wu-sheng added the core feature Core and important feature. Sometimes, break backwards compatibility. label Mar 31, 2020

wu-sheng added this to the 8.0.0 milestone Mar 31, 2020

wu-sheng requested a review from a team March 31, 2020 15:35

1. Set up traffic for service and service instance. 2. Add ServiceIns…

45220fe

…tanceProperties as a new source. 3. Remove Month as downsamping and query step.

wu-sheng mentioned this pull request Apr 1, 2020

Document file name and title are inconsistent #4600

Closed

wu-sheng added 5 commits April 1, 2020 12:39

Merge branch 'master' into v8-core

f9b3b61

Finish all new traffic objects and network alias metrics. Being to ad…

a5e258f

…just the receiver, starting with trace analysis.

Merge branch 'v8-core' of https://github.com/apache/skywalking into v…

5509edb

…8-core

Set up network address alias and fix some listeners.

469e75a

Enhance protocol documentations.

d40ecd3

wu-sheng added 6 commits April 2, 2020 07:35

Update protocol

78ca5f4

Remove all register related interfaces and DAOs. Will try to implemen…

8a6813e

…t them later.

Make topology query pass compling

fe99646

Make core compiling passed.

bda811c

Make more modules compiling passed.

b84bb39

More backend modules compiling passed. Several left.

9abd257

mrproliu mentioned this pull request Apr 4, 2020

Adapt the new v3 protocol apache/skywalking-nginx-lua#30

Merged

mvn clean compile passed!

3afa320

wu-sheng added 6 commits April 4, 2020 18:05

Push more codes.

16c8a78

Package succesfuly with UT passed.

9975943

Make project bootstrap successfully.

ce52703

Fix format.

bc1f8dc

Make analysis and ES persistent good.

37ac99f

Fix model format.

821c4a9

wu-sheng and others added 4 commits April 9, 2020 16:14

1. Support un-tracing when backend is unavailable.

9ec9883

2. Add MeshDataMock.

Support KEEP_TRACING config in agent. And fix the test cases.

aea5940

Fix tests passed by activating KEEP_TRACING.

91a1e9f

Remove month level verification of TTL

6bedce2

kezhenxu94 marked this pull request as ready for review April 9, 2020 11:04

wu-sheng changed the title ~~[WIP] 8.0.0 Core Changes~~ Make 8.0.0 Core available. New protocol and register removed. Apr 9, 2020

Some miss changed docs

fa1702a

wu-sheng commented Apr 9, 2020

View reviewed changes

mrproliu previously approved these changes Apr 9, 2020

View reviewed changes

dmsolr previously approved these changes Apr 9, 2020

View reviewed changes

Update document and configuration to follow v8

0b637a3

wu-sheng dismissed stale reviews from dmsolr and mrproliu via 0b637a3 April 10, 2020 02:09

kezhenxu94 previously approved these changes Apr 10, 2020

View reviewed changes

Fix a typo

8f6fa9f

wu-sheng dismissed kezhenxu94’s stale review via 8f6fa9f April 10, 2020 02:18

kezhenxu94 approved these changes Apr 10, 2020

View reviewed changes

wu-sheng mentioned this pull request Apr 10, 2020

Update the document of agent test. #4628

Closed

hanahmily approved these changes Apr 10, 2020

View reviewed changes

heyanlong approved these changes Apr 10, 2020

View reviewed changes

rainbend approved these changes Apr 10, 2020

View reviewed changes

wu-sheng merged commit 6fe2041 into master Apr 10, 2020

wu-sheng deleted the v8-core branch April 10, 2020 03:03

dmsolr mentioned this pull request Apr 12, 2020

Upgrade the InfluxDB storage-plugin to protocol V3 #4641

Merged

3 tasks

snakorse mentioned this pull request Aug 5, 2020

Support skywalking v8 protocol SkyAPM/SkyAPM-dotnet#313

Closed

4 tasks

Conversation

wu-sheng commented Mar 31, 2020

Uh oh!

wu-sheng commented Apr 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wu-sheng commented Apr 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wu-sheng commented Apr 4, 2020

Uh oh!

kezhenxu94 commented Apr 9, 2020

Uh oh!

wu-sheng Apr 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wu-sheng commented Apr 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

mrproliu left a comment

Choose a reason for hiding this comment

Uh oh!

dmsolr left a comment

Choose a reason for hiding this comment

Uh oh!

wu-sheng commented Apr 10, 2020

Uh oh!

kezhenxu94 commented Apr 10, 2020

Uh oh!

wu-sheng commented Apr 10, 2020

Uh oh!

hanahmily left a comment

Choose a reason for hiding this comment

Uh oh!

heyanlong left a comment

Choose a reason for hiding this comment

Uh oh!

rainbend left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wu-sheng commented Apr 1, 2020 •

edited

Loading

wu-sheng commented Apr 1, 2020 •

edited

Loading

wu-sheng Apr 9, 2020 •

edited

Loading

wu-sheng commented Apr 9, 2020 •

edited

Loading