Support select replica with rich meta data #171

sunxiaoguang · 2021-05-19T09:30:33Z

Signed-off-by: Xiaoguang Sun sunxiaoguang@zhihu.com

What problem does this PR solve?

Pass in more meta data so ReplicaSelector can make better decision at runtime

What is changed and how it works?

Pass in more meta data so ReplicaSelector can make better decision at runtime

Check List

Tests

Unit test

Code changes

Has exported function/method change
Has interface methods change

Side effects

Increased code complexity
Breaking backward compatibility

Related changes

Need to update the documentation
Need to be included in the release note

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

src/main/java/org/tikv/common/PDClient.java

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

…_github

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang · 2021-06-12T13:16:57Z

Please be aware, In addition to implement feature requested in issue #193, there is an extra change to support custom HostMapping implementation to make it possible to map cluster local names in K8S cluster into global names that are accessible out side of K8S cluster.

sunxiaoguang · 2021-06-12T13:25:56Z

Please be aware, In addition to implement feature requested in issue #193, there is an extra change to support custom HostMapping implementation to make it possible to map cluster local names in K8S cluster into global names that are accessible out side of K8S cluster.

There is an example of HostMapping implementation here: https://github.com/tidb-incubator/TiBigData/blob/662469a66ab951fcdf557ab461fceaa90a27b595/tidb/src/main/java/io/tidb/bigdata/tidb/DnsSearchHostMapping.java

sunxiaoguang · 2021-06-13T03:13:25Z

Changes to ReplicaSelector interface made this PR binary incompatible with existing code. Even though I doubt if this feature was actually used anywhere, we can still support the old interface by some extra amount of work. Would like to hear feedbacks.

src/main/java/org/tikv/common/DefaultHostMapping.java

marsishandsome

LGTM

marsishandsome · 2021-06-15T02:50:20Z

/run-all-tests

marsishandsome · 2021-06-15T02:55:17Z

@sunxiaoguang test failed.

[2021-06-15T02:53:55.007Z] [ERROR] /home/jenkins/agent/git/client-java/src/test/java/org/tikv/common/MockServerTest.java:[32,13] cannot find symbol

[2021-06-15T02:53:55.007Z]   symbol:   method of(org.tikv.kvproto.Metapb.Store)

[2021-06-15T02:53:55.007Z]   location: interface java.util.List

sunxiaoguang · 2021-06-15T04:10:37Z

@sunxiaoguang test failed.

[2021-06-15T02:53:55.007Z] [ERROR] /home/jenkins/agent/git/client-java/src/test/java/org/tikv/common/MockServerTest.java:[32,13] cannot find symbol

[2021-06-15T02:53:55.007Z]   symbol:   method of(org.tikv.kvproto.Metapb.Store)

[2021-06-15T02:53:55.007Z]   location: interface java.util.List

Sure, let me fix it.

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang · 2021-06-15T04:25:20Z

/run-all-tests

sunxiaoguang · 2021-06-15T04:42:31Z

/run-all-tests

sunxiaoguang · 2021-06-17T01:52:51Z

/run-all-tests

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

…_github

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

…_github

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang · 2021-06-23T04:42:30Z

/run-all-tests

marsishandsome

LGTM

birdstorm

LGTM

sunxiaoguang · 2021-06-23T07:41:32Z

/merge

ti-srebot · 2021-06-23T07:43:50Z

/run-all-tests

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-06-30T21:07:15Z

cherry pick to release-3.1 in PR #225

* cherry pick #171 to release-3.1 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix test Signed-off-by: birdstorm <samuelwyf@hotmail.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> Co-authored-by: birdstorm <samuelwyf@hotmail.com>

sunxiaoguang changed the title ~~Support select replica with rich meta data~~ [DNM] Support select replica with rich meta data May 19, 2021

Support select replica with rich meta data

bfca20b

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang force-pushed the rich_replica_select branch from 353571f to bfca20b Compare May 19, 2021 09:36

Use string helper

a3b56b0

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

marsishandsome reviewed May 21, 2021

View reviewed changes

src/main/java/org/tikv/common/PDClient.java Show resolved Hide resolved

sunxiaoguang added 3 commits May 26, 2021 21:39

Print debug log when failed to get member

42f10cd

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Merge remote-tracking branch 'origin/master' into rich_replica_select…

fb10732

…_github

Support customized HostMapping implementation

de050bf

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang changed the title ~~[DNM] Support select replica with rich meta data~~ Support select replica with rich meta data Jun 12, 2021

tisonkun reviewed Jun 13, 2021

View reviewed changes

src/main/java/org/tikv/common/DefaultHostMapping.java Show resolved Hide resolved

marsishandsome approved these changes Jun 15, 2021

View reviewed changes

ti-srebot added the status/LGT1 label Jun 15, 2021

marsishandsome requested a review from birdstorm June 15, 2021 02:52

Use ImmutableList as List.of only exists since Java 9

58668af

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang added 7 commits June 17, 2021 20:39

Use this.leader in TiRegion

12512f0

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Fix couple of integration tests

0171bc9

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Fix one more integration test

26a8f53

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Fix replica read peer mismatch

6336492

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Use passed in peer as it already selected peer

ecb8af1

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Merge remote-tracking branch 'origin/master' into rich_replica_select…

6697a99

…_github

Refactor after merge master

0fe131e

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

sunxiaoguang added 3 commits June 23, 2021 11:25

Merge remote-tracking branch 'origin/master' into rich_replica_select…

7ebaa55

…_github

Try fix integration test

c73df3c

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

Workaround lint issue with duplicated dependency for different scopes

2a63421

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

marsishandsome approved these changes Jun 23, 2021

View reviewed changes

birdstorm approved these changes Jun 23, 2021

View reviewed changes

ti-srebot added status/LGT2 and removed status/LGT1 labels Jun 23, 2021

ti-srebot approved these changes Jun 23, 2021

View reviewed changes

ti-srebot added the status/can-merge label Jun 23, 2021

Merge branch 'master' into rich_replica_select

f418882

ti-srebot merged commit fe6617e into tikv:master Jun 23, 2021

birdstorm added the needs-cherry-pick-3.1 label Jun 30, 2021

ti-srebot pushed a commit to ti-srebot/client-java that referenced this pull request Jun 30, 2021

cherry pick tikv#171 to release-3.1

24229f1

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jun 30, 2021

Support select replica with rich meta data (#171) #225

Merged

Daemonxiao mentioned this pull request Jun 30, 2022

Support host mapping in TiSpark pingcap/tispark#2436

Merged

shiyuhang0 mentioned this pull request Sep 5, 2022

Support follower read pingcap/tispark#2534

Closed

Support select replica with rich meta data #171

Support select replica with rich meta data #171

Uh oh!

Conversation

sunxiaoguang commented May 19, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Uh oh!

Uh oh!

sunxiaoguang commented Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunxiaoguang commented Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunxiaoguang commented Jun 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

marsishandsome left a comment

Choose a reason for hiding this comment

Uh oh!

marsishandsome commented Jun 15, 2021

Uh oh!

marsishandsome commented Jun 15, 2021

Uh oh!

sunxiaoguang commented Jun 15, 2021

Uh oh!

sunxiaoguang commented Jun 15, 2021

Uh oh!

sunxiaoguang commented Jun 15, 2021

Uh oh!

sunxiaoguang commented Jun 17, 2021

Uh oh!

sunxiaoguang commented Jun 23, 2021

Uh oh!

marsishandsome left a comment

Choose a reason for hiding this comment

Uh oh!

birdstorm left a comment

Choose a reason for hiding this comment

Uh oh!

sunxiaoguang commented Jun 23, 2021

Uh oh!

ti-srebot commented Jun 23, 2021

Uh oh!

ti-srebot commented Jun 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sunxiaoguang commented Jun 12, 2021 •

edited

Loading

sunxiaoguang commented Jun 12, 2021 •

edited

Loading

sunxiaoguang commented Jun 13, 2021 •

edited

Loading