Skip to content

route single partition refresh #247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
shenyunlong merged 47 commits into from
Jan 3, 2025
Merged

route single partition refresh #247

shenyunlong merged 47 commits into from
Jan 3, 2025

Conversation

@maochongxin
Copy link
Contributor

@maochongxin maochongxin commented Nov 27, 2024 •
edited
Loading

Summary

This feature primarily implements a single-tablet refresh mechanism for table location and adds some retry mechanisms to support the auto-splitting feature.

Solution Description

  • The table location management in 4.x is separated and managed independently using ObPartitionLocationInfo, with index management based on tablet IDs in ObPartitionEntry.
  • The refresh of metadata and the refresh of location are decoupled into two phases. When a routing change is encountered, the location information is refreshed at a single-tablet granularity.
  • A retry mechanism has been added to refresh the entire table entry for certain error codes and appropriately handle requests that require retries (e.g., splitting and recombining batches) to ensure the reliability and correctness of the retries.

@maochongxin maochongxin force-pushed the feature/route-single-partition-refresh branch 2 times, most recently from cb8be8e to 73569e8 Compare December 10, 2024 11:13
@maochongxin maochongxin force-pushed the feature/route-single-partition-refresh branch from c8b1682 to 0954f28 Compare December 17, 2024 02:44
@maochongxin maochongxin force-pushed the feature/route-single-partition-refresh branch 2 times, most recently from b7f625a to 0899fea Compare January 2, 2025 09:15
maochongxin and others added 25 commits January 3, 2025 15:16
@maochongxin
Optimize SQL for refreshing table location information
75c393f
@maochongxin
partical refresh
63a5ebb
@maochongxin
Fix infinite loop caused by removed reference
19b2d55
@maochongxin
fix lsop retry fail
1149af4
@maochongxin
Remove unnecessary comments and format code
a8cbdad
@maochongxin
Fix frequent refresh lock failures due to short refresh interval
b2ffaec
@maochongxin
Fix frequent refresh lock failures due to short refresh interval
21f89ff
@maochongxin
add result code -4723
5af5ecc
@maochongxin
fix review
756b48c
@maochongxin
add -4138
c2ddf78
@maochongxin
fix review: add ut for byteutil
b59b05d
@maochongxin
fix lsop refresh location
491a7b1
@maochongxin
Merge pull request #204 from oceanbase/retry_batchops_merge_master
19ca283 
Enhance Client Support for Partition Splitting
@maochongxin
feature: Optimize partial refresh and add retry for executeMutation/q…
9ddfc8d 
…uery (#213)

* remove lock in refreshTableLocationByTabletId

* fix refresh sql

* fix refresh interval

* add retry logic for common query

* add retry logic for executeMutation

* add retry logic for executeMutation

* add retry logic for ObTableQueryAndMutateRequest

* fix: correct ineffective retry logic

* retry logic for ObTableClient execute

* fix

* fix
@miyuan-ljr @maochongxin
fix 3.x null exception (#214)
1ee0be6 
* fix

* fix regress
@maochongxin @shenyunlong
[fix] add threshold for refresh table entry with location (#220)
5a52c0d 
* add threshold for refresh table entry with location

* [Fix] remove TABLE_ENTRY_LOCATION_REFRESH_THRESHOLD in Property

* [Fix] fix compile error

---------

Co-authored-by: shenyunlong.syl <shenyunlong.syl@oceanbase.com>
@miyuan-ljr @maochongxin
[fix] 3.x compatible (#223)
1a47442 
* fix

* fix
@miyuan-ljr @maochongxin
[fix] test case stuck in await (#227)
17107a8 
* fix

* fix

* refine
@GroundWu @maochongxin
Fix global route refresh wrong (#231)
6920d74 
* fix global index route refresh wrong when route need refresh

* fix global index route wrong

* revert unused commit
@GroundWu @maochongxin
fix global index route wrong when need partial route refresh (#232)
c803c3f 
* fix global index route wrong when need partial route refresh

* Ensure PartitionInfo is Exposed Only After Leader is Found During Single Shard Refresh

---------

Co-authored-by: maochongxin <maochongxin@gmail.com>
@maochongxin
fix PartitionNumOneTest routed to the wrong server
6ce34c4
@maochongxin
add error code 5627
112e484
@maochongxin
fix extendType parse object to comparable
24f0ca2
@maochongxin
fix refresh location param
ba5fa12
@maochongxin
fix log; add retry interval for executeWithRetries
c234aa7
@maochongxin
Add quick path: return quickly if recently refreshed
62181d1
@maochongxin
add log for batchOpsImpl
8739c22
@maochongxin
fix
1316564
@maochongxin
fix table.entry.refresh.interval.wait
9d3d75e
@maochongxin
fix global index route error
8773f5d
@maochongxin
fix
5205bcc
@maochongxin
TABLE_ENTRY_LOCATION_REFRESH_THRESHOLD -> 0
86a6ac1
@maochongxin
add retry code =4242
eb260ad
@maochongxin
fix getPartitionLocation nullptr && resolve conflict error
4dc8ab0
@maochongxin
fix resolve conflict error
b2383d9
@maochongxin
fix getLs nullptr
8beb8de
@maochongxin
fix dr nullptr
ea257f8
@maochongxin
fix global index nullptr
932b67a
@maochongxin
fix unexpected addr expired
50c1ee9
@maochongxin
fix unexpected addr expired
af5a620
@maochongxin
Reduce log printing and optimize log output content;
8ef2267
@maochongxin
fix update lsId if location exists
c36e75f
@maochongxin
fix log
ad01c8b
@maochongxin
fix lsop -5200 with table_group
091f741
@maochongxin
fix rpc.execute.timeout
f12db21
@maochongxin
fix log
baaa435
@maochongxin
fix log && fix getPartitionReplica return wrong part id
83312c3
@maochongxin maochongxin force-pushed the feature/route-single-partition-refresh branch from 0899fea to 83312c3 Compare January 3, 2025 07:16
@shenyunlong
Copy link
Contributor

shenyunlong commented Jan 3, 2025

LGTM

@shenyunlong shenyunlong merged commit bd246ea into master Jan 3, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@maochongxin @shenyunlong @miyuan-ljr @GroundWu