Update RFC of API v2 according to latest codes #79

pingyu · 2021-10-26T12:43:55Z

Signed-off-by: pingyu yuping@pingcap.com

Update RFC of API v2 according to latest codes.

Rendered: https://github.com/pingyu/tikv-rfcs/blob/rfc_api_v2_pd/text/0069-api-v2.md

pingyu · 2021-10-26T12:45:15Z

iosmanthus · 2021-10-27T07:31:59Z

text/0069-api-v2.md

+
+3. Add new keyspace.
+```sh
+➜ txn: cmp: create("/keyspaces/feature-store") = "0", then put /keyspaces/feature-store {"id":1}


This could be summarized as compare-and-swap?

What if this "Txn" is failed? Do we need to return an error or retry in server-side with fixed rounds?

I think it would be better to write down more detail about how to implement the compare-and-swap here, even it seems to be too verbose.

etcd doesn't provide an CAS interface actually, but a transaction interface. And I'm not very familiar to etcd. So I think more detail would help entirely confirming the correctness of this design.

What if this "Txn" is failed? Do we need to return an error or retry in server-side with fixed rounds?

Return of Txn.Commit() will tell us success, transaction fail caused by conflict, or error occurred.
On transaction conflict, we can retry limited times (e.g. 3), return TransactionConflict to user if exceed.
On error occurred, report the error to user.

iosmanthus · 2021-10-27T09:47:38Z

text/0069-api-v2.md

+➜ txn: cmp ver("/keyspaces/feature-store") = "1" then put /keyspaces_deleted/1_feature-store {"id":1}, del /keyspaces/feature-store
+```
+
+#### Flashback


Could I comprehend this as a kind of restore operation?

Yes, undo a delete by mistake.

andylokandy · 2021-12-27T07:56:20Z

@pingyu I've update the rfc, PTAL

pingyu · 2021-12-27T09:20:59Z

text/0069-api-v2.md


->> config keyspaces delete-property <keyspace name> <property-path>
->> config keyspaces flashback <new keyspace name>
+>> config keyspaces create <keyspace name> [--id <Keyspace ID>]


Is there any restriction on the provided Keyspace ID ? For example, what if the ID is the same as another Keyspace ?

What if two concurrent create requests with different name but same id ?

AFAIK etcd provides transaction API similar to CAS. It seems to be difficult to solve such a race condition.

pingyu · 2021-12-27T09:31:47Z

text/0069-api-v2.md

->> config keyspaces flashback <new keyspace name>
+>> config keyspaces create <keyspace name> [--id <Keyspace ID>]
+>> config keyspaces delete <keyspace name>
+>> config keyspaces flashback [<new keyspace name>]


Suggest describe how flashback is implemented.
Besides, flashback will delete the entry in /keyspaces_deleted (won't it ?), but PD doesn't provide such an API.

updated, PTAL

It is possible to be conflicted with another concurrent create request with 1) same name, or 2) different name but same id.

Etcd's transaction API can help to deal with the same name condition. But it would be difficult to solve the same id case.

I think we allow multiple keyspace names referring to the same id.

I think it would be better not to allow this.

Users would assume that two keyspaces are different if they have different names. Especially when two keyspaces have same id silently caused by the race condition. The two keyspaces will corrupt each other.

pingyu · 2022-11-02T05:31:44Z

@iosmanthus @haojinming @zeminzhou PTAL, thanks~

haojinming

LGTM

ystaticy · 2022-11-02T06:56:48Z

LGTM

text/0069-api-v2.md

Signed-off-by: pingyu <yuping@pingcap.com>

* rawkv bulk load: add description for pause merge Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Update text/0072-online-bulk-load-for-rawkv.md Co-authored-by: Liangliang Gu <marsishandsome@gmail.com> Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Add future improvements Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Liangliang Gu <marsishandsome@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: Andy Lok <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: Andy Lok <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

* RFC: In-memory Pessimistic Locks Signed-off-by: Yilin Chen <sticnarf@gmail.com> * clarify where to delete memory locks after writing a lock CF KV Signed-off-by: Yilin Chen <sticnarf@gmail.com> * Elaborate transfer leader handlings and add correctness section Signed-off-by: Yilin Chen <sticnarf@gmail.com> * add an addition step of proposing pessimistic locks before transferring leader Signed-off-by: Yilin Chen <sticnarf@gmail.com> * clarify about new leaders of region split Signed-off-by: Yilin Chen <sticnarf@gmail.com> * Add tracking issue link Signed-off-by: Yilin Chen <sticnarf@gmail.com> * update design and correctness analysis of lock migration Signed-off-by: Yilin Chen <sticnarf@gmail.com> * add configurations Signed-off-by: Yilin Chen <sticnarf@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: pingyu <yuping@pingcap.com>

pingyu · 2022-11-04T10:22:29Z

@sunxiaoguang All comments are addressed (see here. The force-push is to fix DCO error), PTAL, thanks~

iosmanthus

LGTM

iosmanthus reviewed Oct 27, 2021

View reviewed changes

andylokandy requested review from BusyJay and nolouch December 27, 2021 09:05

pingyu commented Dec 27, 2021

View reviewed changes

andylokandy mentioned this pull request Feb 23, 2022

Update API V2 #75

Closed

andylokandy changed the title ~~docs: Implementation detail of PD for API v2~~ docs: Implementation detail of API v2 Feb 23, 2022

This was referenced Feb 28, 2022

api_version: Codec for RawKV key tikv/tikv#12036

Merged

causal_ts: Add causal timestamp component for RawKV CDC tikv/tikv#12072

Closed

This was referenced Mar 8, 2022

storage: Modify rawkv interfaces for API V2 encoding tikv/tikv#12128

Closed

rawkv: causal timestamp tikv/tikv#12099

Merged

pingyu mentioned this pull request Apr 3, 2022

storage: add APIVersion as generic parameter tikv/tikv#12324

Merged

pingyu changed the title ~~docs: Implementation detail of API v2~~ Update RFC of API v2 according to latest codes Nov 2, 2022

pingyu requested a review from haojinming November 2, 2022 05:31

haojinming approved these changes Nov 2, 2022

View reviewed changes

sunxiaoguang reviewed Nov 4, 2022

View reviewed changes

text/0069-api-v2.md Outdated Show resolved Hide resolved

text/0069-api-v2.md Outdated Show resolved Hide resolved

text/0069-api-v2.md Outdated Show resolved Hide resolved

text/0069-api-v2.md Outdated Show resolved Hide resolved

text/0069-api-v2.md Outdated Show resolved Hide resolved

marsishandsome and others added 9 commits November 4, 2022 18:18

RFC: RawKV Batch Export (tikv#76)

c91eea4

Signed-off-by: pingyu <yuping@pingcap.com>

ref pd#4112: implementation detail of PD

64b0fec

Signed-off-by: pingyu <yuping@pingcap.com>

ref pd#4112: implementation detail of PD

b91cfcf

Signed-off-by: pingyu <yuping@pingcap.com>

remove raw cf

401308b

Signed-off-by: Andy Lok <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

update

3c1f765

Signed-off-by: Andy Lok <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

update pd design

d294143

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

revert to keyspace_next_id

4b0aa86

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

RFC: Improve the Scalability of TSO Service (tikv#78)

82401e9

Signed-off-by: pingyu <yuping@pingcap.com>

BusyJay and others added 13 commits November 4, 2022 18:18

make region size dynamic (tikv#82)

242df0e

Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

update pd url

c6559be

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

address comment

536928d

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

resolve pd flashback problem

e1751ad

Signed-off-by: andylokandy <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

update rfcs

6794e8a

Signed-off-by: Andy Lok <andylokandy@hotmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

propose online unsafe recovery (tikv#91)

d014d3e

Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

physical isolation between region (tikv#93)

88fff35

Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Signed-off-by: pingyu <yuping@pingcap.com>

wip

3335cd0

Signed-off-by: pingyu <yuping@pingcap.com>

update

3c57572

Signed-off-by: pingyu <yuping@pingcap.com>

update

ca867b2

Signed-off-by: pingyu <yuping@pingcap.com>

Apply suggestions from code review

0cdfbe3

Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> Signed-off-by: pingyu <yuping@pingcap.com>

fix case

dc977ba

Signed-off-by: pingyu <yuping@pingcap.com>

pingyu force-pushed the rfc_api_v2_pd branch from 27f7de1 to dc977ba Compare November 4, 2022 10:19

Merge branch 'master' into rfc_api_v2_pd

763b5c6

pingyu mentioned this pull request Nov 7, 2022

Add GC part to RFC of API v2 #103

Merged

sunxiaoguang approved these changes Nov 8, 2022

View reviewed changes

iosmanthus approved these changes Nov 8, 2022

View reviewed changes

pingyu merged commit b1a22d7 into tikv:master Nov 8, 2022

Update RFC of API v2 according to latest codes #79

Update RFC of API v2 according to latest codes #79

Uh oh!

Conversation

pingyu commented Oct 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pingyu commented Oct 26, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andylokandy commented Dec 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andylokandy Dec 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pingyu commented Nov 2, 2022

Uh oh!

haojinming left a comment

Choose a reason for hiding this comment

Uh oh!

ystaticy commented Nov 2, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pingyu commented Nov 4, 2022

Uh oh!

iosmanthus left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

pingyu commented Oct 26, 2021 •

edited

Loading

andylokandy Dec 29, 2021 •

edited

Loading