RATIS-1894. Implement ReadOnly based on leader lease #925

SzyWilliam · 2023-09-24T02:18:00Z

see https://issues.apache.org/jira/browse/RATIS-1894

OneSizeFitsQuorum

A great job is coming to an end!

I have left some minor issue~ PTAL

OneSizeFitsQuorum · 2023-09-25T03:23:25Z

ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderStateImpl.java

   * @return current readIndex.
   */
-  CompletableFuture<Long> getReadIndex(Long readAfterWriteConsistentIndex) {
+  CompletableFuture<Long> getReadIndex(RaftServerConfigKeys.Read.Option option, Long readAfterWriteConsistentIndex) {


It seems that with lease read we need to change the name of this function (and maybe some other functions on the call stack). The important logic for both readIndex and leaseRead is three steps.

Verify that the current node is the Leader (readIndex uses a logical clock to initiate heartbeat, lease read uses a physical clock with an error upper bound),

Wait for the local applyIndex to reach the current commitIndex

Query the state machine.

Maybe we could name the first step confirmLeader or something

They both need to confirm leader and obtain readIndex first. ReadOnly(ReadIndex based) uses majority heartbeats to obtain the readIndex while ReadOnly(Lease based) uses leader lease. I think readIndex is a shared step

Got it! This really depends on how to understand the meaning of the word readIndex. If we consider that the follower also needs to get commitIndex from the leader when using lease read, it is reasonable for me to support current naming logic, so please ignore this review~

OneSizeFitsQuorum · 2023-09-25T03:38:15Z

ratis-server-api/src/main/java/org/apache/ratis/server/RaftServerConfigKeys.java

+      LINEARIZABLE,
+
+      /** Use Leader lease. Efficient than ReadIndex if clock drift is bounded */
+      LEASE


It seems that both DEFAULT and LINEARIZABLE are talking about consistency issues. Lease and ReadIndex are implementation techniques for implementing LINEARIZABLE, so it seems inappropriate to mention lease alone alongside the other two options?

You are right. DEFAULT / LINEARIZABLE / LEASE sound vague to the terminal users. We should consider renaming these options.

Is there a good reason to use LINEARIZABLE not LEASE?

If yes, we could have a separated configuration to enable/disable LeaderLease.

If not, we could just have LINEARIZABLE for ReadIndex + LeaderLease.

+1 Patch looks good other that this. Thanks @SzyWilliam for working on this!

OneSizeFitsQuorum · 2023-09-26T02:50:16Z

ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderLease.java

  LeaderLease(RaftProperties properties) {
+    this.enabled = RaftServerConfigKeys.Read.leaderLeaseEnabled(properties);
    final double leaseRatio = RaftServerConfigKeys.Read.leaderLeaseTimeoutRatio(properties);
    Preconditions.assertTrue(leaseRatio > 0.0 && leaseRatio <= 1.0,


do we need to check this when lease read is disabled?

OneSizeFitsQuorum · 2023-09-26T06:27:23Z

ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderLease.java


-  boolean isEnabled() {
-    return enabled;
+  boolean getAndSetEnabled(boolean enabled) {


when you set this field to true?

It looks like now it won't be true as long as it's set to false once

It will be set back to original value if we failed transferring leadership. In other situations (stepDown / successful transferring), the LeaderStateImpl will be stopped and the enabled will stay in false.

I see, the new leader will create a new lease according to the configuration parameters~

SzyWilliam · 2023-09-26T06:36:04Z

Is there a good reason to use LINEARIZABLE not LEASE?

I believe we have.

LINEARIZABLE should meet most requirements in typical scenarios. It allows followers to handle read requests while maintaining strict linearizability. This will be the default and recommended option to use in IoTDB.

LEASE is designed for situations where low latency is crucial, even at the potential cost of linearizability violations. The use of LEASE is mandated by Alluxio @codings-dan. His experiments have demonstrated that in a 3-node cluster, using LINEARIZABLE alone results in only a 160% improvement in throughput due to latency, whereas theoretically, it could achieve a 300% improvement.

Nevertheless, IoTDB may consider employing this option if the potential benefits outweigh the drawbacks.

Also, I added revoke mechanisms during leader-transfer / stepDown. PTAL, thanks!
@szetszwo @OneSizeFitsQuorum

OneSizeFitsQuorum · 2023-09-27T02:25:00Z

ratis-server/src/main/java/org/apache/ratis/server/impl/LeaderStateImpl.java

    }

+    // if lease is enabled, check lease first
+    if (hasLease()) {


It seems that even though lease read is not enabled, we are still calling hasLease frequently and attempting to extend lease many times. Maybe we could use some optimization here, since it's a critical step in the query.

You are right! Fixed, PTAL.

szetszwo

+1 the change looks good.

OneSizeFitsQuorum

LGTM~
Thanks a lot for @SzyWilliam! Lease read is expected to be available to more users in ratis 3.0!

### What changes were proposed in this pull request? Bump Ratis version from 2.5.1 to 3.0.1. Address incompatible changes: - RATIS-589. Eliminate buffer copying in SegmentedRaftLogOutputStream.(apache/ratis#964) - RATIS-1677. Do not auto format RaftStorage in RECOVER.(apache/ratis#718) - RATIS-1710. Refactor metrics api and implementation to separated modules. (apache/ratis#749) ### Why are the changes needed? Bump Ratis version from 2.5.1 to 3.0.1. Ratis has released v3.0.0, v3.0.1, which release note refers to [3.0.0](https://ratis.apache.org/post/3.0.0.html), [3.0.1](https://ratis.apache.org/post/3.0.1.html). The 3.0.x version include new features like pluggable metrics and lease read, etc, some improvements and bugfixes including: - 3.0.0: Change list of ratis 3.0.0 In total, there are roughly 100 commits diffing from 2.5.1 including: - Incompatible Changes - RaftStorage Auto-Format - RATIS-1677. Do not auto format RaftStorage in RECOVER. (apache/ratis#718) - RATIS-1694. Fix the compatibility issue of RATIS-1677. (apache/ratis#731) - RATIS-1871. Auto format RaftStorage when there is only one directory configured. (apache/ratis#903) - Pluggable Ratis-Metrics (RATIS-1688) - RATIS-1689. Remove the use of the thirdparty Gauge. (apache/ratis#728) - RATIS-1692. Remove the use of the thirdparty Counter. (apache/ratis#732) - RATIS-1693. Remove the use of the thirdparty Timer. (apache/ratis#734) - RATIS-1703. Move MetricsReporting and JvmMetrics to impl. (apache/ratis#741) - RATIS-1704. Fix SuppressWarnings(“VisibilityModifier”) in RatisMetrics. (apache/ratis#742) - RATIS-1710. Refactor metrics api and implementation to separated modules. (apache/ratis#749) - RATIS-1712. Add a dropwizard 3 implementation of ratis-metrics-api. (apache/ratis#751) - RATIS-1391. Update library dropwizard.metrics version to 4.x (apache/ratis#632) - RATIS-1601. Use the shaded dropwizard metrics and remove the dependency (apache/ratis#671) - Streaming Protocol Change - RATIS-1569. Move the asyncRpcApi.sendForward(..) call to the client side. (apache/ratis#635) - New Features - Leader Lease (RATIS-1864) - RATIS-1865. Add leader lease bound ratio configuration (apache/ratis#897) - RATIS-1866. Maintain leader lease after AppendEntries (apache/ratis#898) - RATIS-1894. Implement ReadOnly based on leader lease (apache/ratis#925) - RATIS-1882. Support read-after-write consistency (apache/ratis#913) - StateMachine API - RATIS-1874. Add notifyLeaderReady function in IStateMachine (apache/ratis#906) - RATIS-1897. Make TransactionContext available in DataApi.write(..). (apache/ratis#930) - New Configuration Properties - RATIS-1862. Add the parameter whether to take Snapshot when stopping to adapt to different services (apache/ratis#896) - RATIS-1930. Add a conf for enable/disable majority-add. (apache/ratis#961) - RATIS-1918. Introduces parameters that separately control the shutdown of RaftServerProxy by JVMPauseMonitor. (apache/ratis#950) - RATIS-1636. Support re-config ratis properties (apache/ratis#800) - RATIS-1860. Add ratis-shell cmd to generate a new raft-meta.conf. (apache/ratis#901) - Improvements & Bug Fixes - Netty - RATIS-1898. Netty should use EpollEventLoopGroup by default (apache/ratis#931) - RATIS-1899. Use EpollEventLoopGroup for Netty Proxies (apache/ratis#932) - RATIS-1921. Shared worker group in WorkerGroupGetter should be closed. (apache/ratis#955) - RATIS-1923. Netty: atomic operations require side-effect-free functions. (apache/ratis#956) - RaftServer - RATIS-1924. Increase the default of raft.server.log.segment.size.max. (apache/ratis#957) - RATIS-1892. Unify the lifetime of the RaftServerProxy thread pool (apache/ratis#923) - RATIS-1889. NoSuchMethodError: RaftServerMetricsImpl.addNumPendingRequestsGauge apache/ratis#922 (apache/ratis#922) - RATIS-761. Handle writeStateMachineData failure in leader. (apache/ratis#927) - RATIS-1902. The snapshot index is set incorrectly in InstallSnapshotReplyProto. (apache/ratis#933) - RATIS-1912. Fix infinity election when perform membership change. (apache/ratis#954) - RATIS-1858. Follower keeps logging first election timeout. (apache/ratis#894) - 3.0.1:This is a bugfix release. See the [changes between 3.0.0 and 3.0.1](apache/ratis@ratis-3.0.0...ratis-3.0.1) releases. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Cluster manual test. Closes #2480 from SteNicholas/CELEBORN-1400. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>

SzyWilliam added 2 commits September 24, 2023 10:12

add leader lease read

786f519

add leader lease read

45e227e

SzyWilliam changed the title ~~Implement ReadOnly based on leader lease~~ RATIS-1894. Implement ReadOnly based on leader lease Sep 24, 2023

add leader lease read

b1d12ec

SzyWilliam requested a review from szetszwo September 25, 2023 02:23

OneSizeFitsQuorum reviewed Sep 25, 2023

View reviewed changes

SzyWilliam added 3 commits September 26, 2023 09:54

address review issues

4202af8

address review issues

6d82680

address review issues

9f91f77

OneSizeFitsQuorum reviewed Sep 26, 2023

View reviewed changes

SzyWilliam added 2 commits September 26, 2023 12:19

revoke lease during transfer or stepdown

9ce652b

make checkstyle happy

f730931

OneSizeFitsQuorum reviewed Sep 26, 2023

View reviewed changes

SzyWilliam added 3 commits September 26, 2023 14:41

make checkstyle happy

85dc009

fix test fails

07b796e

fix test fails

505ce9f

OneSizeFitsQuorum reviewed Sep 27, 2023

View reviewed changes

address review issues

8055787

szetszwo approved these changes Sep 27, 2023

View reviewed changes

OneSizeFitsQuorum approved these changes Sep 28, 2023

View reviewed changes

SzyWilliam merged commit 0711c39 into apache:feature/leaderlease Sep 28, 2023

SteNicholas mentioned this pull request Apr 24, 2024

[CELEBORN-1400] Bump Ratis version from 2.5.1 to 3.0.1 apache/celeborn#2480

Closed

RATIS-1894. Implement ReadOnly based on leader lease #925

RATIS-1894. Implement ReadOnly based on leader lease #925

Uh oh!

Conversation

SzyWilliam commented Sep 24, 2023

Uh oh!

OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OneSizeFitsQuorum Sep 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SzyWilliam commented Sep 26, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

OneSizeFitsQuorum Sep 26, 2023 •

edited

Loading